Sep 16 18:41:34.769314 ip-10-0-128-180 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:41:34.769324 ip-10-0-128-180 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:41:34.769331 ip-10-0-128-180 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:41:34.769543 ip-10-0-128-180 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:41:44.278338 ip-10-0-128-180 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 16 18:41:44.278351 ip-10-0-128-180 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f4970c50e00e47e8af76620cc302a7a7 --
Sep 16 18:43:17.479607 ip-10-0-128-180 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:43:17.909807 ip-10-0-128-180 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:17.909807 ip-10-0-128-180 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:43:17.909807 ip-10-0-128-180 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:17.909807 ip-10-0-128-180 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:43:17.909807 ip-10-0-128-180 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:17.910529 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.909899    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:43:17.915396 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915379    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:17.915396 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915396    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915399    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915403    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915406    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915409    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915412    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915415    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915418    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915421    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915423    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915426    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915429    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915431    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915435    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915437    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915440    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915442    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915445    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915448    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:17.915458 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915451    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915453    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915456    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915459    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915462    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915465    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915468    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915471    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915474    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915476    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915479    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915481    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915484    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915486    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915489    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915491    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915495    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915497    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915500    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915502    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:17.915899 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915505    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915508    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915510    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915514    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915518    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915521    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915525    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915528    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915531    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915534    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915537    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915540    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915543    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915546    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915548    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915551    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915553    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915556    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915559    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:17.916391 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915564    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915573    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915576    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915579    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915582    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915584    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915587    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915590    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915592    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915595    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915597    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915623    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915627    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915630    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915633    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915636    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915638    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915641    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915644    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915647    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:17.916860 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915649    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915652    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915654    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915657    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915661    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915663    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915666    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915668    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915671    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915674    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915676    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915679    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915681    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915684    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915687    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915690    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915693    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915695    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915698    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.915700    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:17.917387 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917258    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917266    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917269    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917272    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917275    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917278    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917280    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917283    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917287    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917291    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917294    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917296    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917298    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917301    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917304    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917307    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917310    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917313    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917330    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917333    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:17.917861 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917335    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917338    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917341    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917343    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917346    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917349    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917351    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917354    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917357    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917361    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917364    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917367    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917370    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917373    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917375    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917378    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917380    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917383    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917385    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:17.918366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917388    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917391    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917393    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917396    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917399    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917401    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917404    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917406    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917410    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917413    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917415    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917419    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917422    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917425    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917427    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917430    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917433    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917435    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917438    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:17.918835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917440    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917442    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917446    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917449    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917451    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917454    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917456    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917459    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917462    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917464    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917466    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917469    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917471    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917474    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917476    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917479    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917481    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917486    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917488    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917491    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:17.919299 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917493    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917496    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917499    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917502    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917504    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917507    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917509    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917511    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917514    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917516    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917519    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917522    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917524    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917527    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917529    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917532    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917534    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917538    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917541    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:17.919798 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917544    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917547    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917913    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917918    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917921    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917924    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917927    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917929    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917932    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917935    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917937    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917940    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917943    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917946    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917949    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917951    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917954    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917957    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917959    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917962    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:17.920257 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917964    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917967    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917970    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917972    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917974    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917977    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917979    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917982    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917985    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917987    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917990    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917992    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917995    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.917997    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918000    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918002    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918005    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918007    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918010    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:17.920761 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918013    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918016    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918019    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918021    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918024    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918026    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918029    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918031    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918034    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918036    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918039    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918041    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918044    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918046    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918049    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918051    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918056    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918059    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918062    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918065    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:17.921220 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918067    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918070    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918073    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918075    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918077    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918080    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918082    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918084    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918087    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918089    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918092    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918095    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918097    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918100    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918103    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918105    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918108    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918110    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918112    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918115    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:17.921784 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918117    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918120    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918122    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918125    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918127    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918130    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918132    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918137    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918140    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918142    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918145    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918147    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918150    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918152    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918155    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918158    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918160    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918164    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918167    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:17.922293 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918170    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918173    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918176    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918262    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918301    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918308    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918312    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918330    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918333    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918337    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918346    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918349    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918352    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918356    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918359    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918362    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918365    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918368    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918371    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918374    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918377    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918383    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918385    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918389    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:43:17.922754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918392    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918396    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918400    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918403    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918406    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918410    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918413    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918415    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918418    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918421    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918424    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918428    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918431    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918433    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918437    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918440    2562 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918442    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918447    2562 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918449    2562 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918452    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918455    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918458    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918462    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918465    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918468    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:43:17.923347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918471    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918474    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918477    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918479    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918482    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918485    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918488    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918490    2562 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918497    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918500    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918503    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918506    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918509    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918515    2562 flags.go:64] FLAG: --help="false"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918518    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-128-180.ec2.internal"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918521    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918524    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918527    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918530    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918534    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918537    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918540    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918542    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:43:17.923940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918545    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918548    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918551    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918553    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918556    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918559    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918562    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918565    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918568    2562 flags.go:64] FLAG: --lock-file=""
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918570    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918573    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918575    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918581    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918583    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918586    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918589    2562 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918592    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918596    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918599    2562 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918602    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918606    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918609    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918613    2562 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918625    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918628    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:43:17.924533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918631    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918634    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918637    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918642    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918645    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918656    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918659    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918662    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918665    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918668    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918673    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918676    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918679    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918681    2562 flags.go:64] FLAG: --port="10250"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918685    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918688    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02676c1e37ea7e882"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918691    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918694    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918697    2562 flags.go:64] FLAG: --register-node="true"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918699    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918703    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918706    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918708    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918711    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918714    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:43:17.925142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918718    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918721    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918724    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918727    2562 flags.go:64] FLAG: --runonce="false"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918729    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918732    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918736    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918739    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918741    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918744    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918753    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918756    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918759    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918762    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918765    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918768    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918771    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918774    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918776    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918781    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918784    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918787    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918790    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918793    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918796    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:43:17.925805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918798    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918801    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918805    2562 flags.go:64] FLAG: --v="2"
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918808    2562 flags.go:64] FLAG: --version="false"
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918812    2562 flags.go:64] FLAG: --vmodule=""
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918816    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.918819    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918931    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918935    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918939    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918942    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918946    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918950    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918954    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918957    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918960    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918963    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918967    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918970    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918973    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:17.926434 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918976    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918978    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918981    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918984    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918986    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918989    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918991    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918994    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918996    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.918999    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919001    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919004    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919006    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919009    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919012    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919014    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919017    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919019    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919022    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919026    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:17.927279 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919029    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919032    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919034    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919037    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919039    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919042    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919044    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919048    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919050    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919054    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919057    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919060    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919062    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919065    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919067    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919070    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919072    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919074    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919077    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919079    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:17.928065 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919082    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919084    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919087    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919090    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919092    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919095    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919097    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919099    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919102    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919104    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919107    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919111    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919113    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919116    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919119    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919121    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919124    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919127    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919129    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919132    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:17.928669 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919135    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919138    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919141    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919145    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919149    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919151    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919154    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919157    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919159    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919162    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919164    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919167    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919169    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919172    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919174    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919177    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919188    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919191    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919193    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919196    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:17.929171 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919199    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919201    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919204    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919207    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919210    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.919213    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.919861    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.929391    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.929407    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.929425    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.929431    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929486    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929490    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929494    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929497    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:17.929702 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929500    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929503    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929507    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929511    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929515    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929518    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929521    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929524    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929527    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929529    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929532    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929535    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929538    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929540    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929543    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929546    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929548    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929551    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929554    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:17.930071 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929556    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929559    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929561    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929564    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929566    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929569    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929573    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929576    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929578    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929581    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929584    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929587    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929591    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929594    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929597    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929600    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929603    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929606    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929609    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929612    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:17.930547 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929615    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929617    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929620    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929623    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929626    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929628    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929631    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929633    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929636    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929638    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929641    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929643    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929646    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929648    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929651    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929653    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929656    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929659    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929662    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:17.931030 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929665    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929668    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929670    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929673    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929676    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929678    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929681    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929684    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929686    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929689    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929691    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929695    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929698    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929701    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929703    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929706    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929709    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929713    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929716    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:17.931509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929718    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929721    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929723    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929726    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929729    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929731    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929734    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929736    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929739    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929742    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929744    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929747    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929750    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929753    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929755    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929758    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929761    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:17.931963 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929764    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.929769    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929887    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929893    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929896    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929899    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929902    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929905    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929908    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929910    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929913    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929916    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929918    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929921    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929924    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:17.932448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929927    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929930    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929932    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929935    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929938    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929940    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929943    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929945    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929948    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929950    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929952    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929955    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929958    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929960    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929963    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929965    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929968    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929972    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929975    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929978    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:17.932835 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929981    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929983    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929986    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929988    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929991    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929993    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929996    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.929999    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930002    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930005    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930007    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930010    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930013    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930015    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930018    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930020    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930023    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930025    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930027    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:17.933363 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930030    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930033    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930037    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930040    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930043    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930047    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930050    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930052    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930055    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930058    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930061    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930064    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930067    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930069    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930072    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930075    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930078    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930081    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930084    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:17.933813 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930086    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930089    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930092    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930094    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930097    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930099    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930102    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930106    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930108    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930111    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930113    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930116    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930118    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930121    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930123    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930126    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930128    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930131    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930133    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930136    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:17.934273 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930139    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930141    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930143    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930146    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930149    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930151    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930155    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:17.930159    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.930164    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.930264    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.932680    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.933688    2562 server.go:1019] "Starting client certificate rotation"
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.933783    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:17.934760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.933812    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:17.936617 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.936604    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:17.937868 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.937848    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:17.953790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.953770    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:43:17.958661 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.958646    2562 log.go:25] "Validated CRI v1 image API"
Sep 16 18:43:17.959576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.959562    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:43:17.963607 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.963589    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:17.965664 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.965646    2562 fs.go:135] Filesystem UUIDs: map[2d25501a-7a44-487a-b45f-43b6eb1d9dba:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a69fcf76-f20c-4a1e-9e56-5f60917a8632:/dev/nvme0n1p4]
Sep 16 18:43:17.965712 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.965666    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:43:17.972614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.972493    2562 manager.go:216] Machine: {Timestamp:2026-09-16 18:43:17.971023925 +0000 UTC m=+0.387726675 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3070074 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f13d7ece66d3a62e9fbda42b35d1a SystemUUID:ec2f13d7-ece6-6d3a-62e9-fbda42b35d1a BootID:f4970c50-e00e-47e8-af76-620cc302a7a7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:05:3b:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:05:3b:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:5a:8e:90:00:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:43:17.972614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.972603    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:43:17.972741 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.972695    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:43:17.976168 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.976142    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:43:17.976297 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.976171    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-180.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:43:17.976986 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.976977    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:43:17.977043 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.976989    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:43:17.977043 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.976997    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:17.977860 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.977850    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:17.979067 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.979057    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:43:17.983407 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983396    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:17.983523 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983514    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:43:17.983561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983544    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:43:17.983561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983554    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:43:17.983614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983567    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:43:17.983614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983577    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:43:17.983614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983586    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:43:17.983938 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983924    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:17.983980 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.983948    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:17.986604 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.986586    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:43:17.989721 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.989708    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:43:17.989763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.989739    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:43:17.991754 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991741    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:43:17.991786 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991764    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:43:17.991786 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991773    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:43:17.991786 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991780    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991789    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991795    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991809    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991814    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991820    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991826    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991834    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991839    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991862    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:43:17.991870 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.991868    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:43:17.996125 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.996095    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-180.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:43:17.996125 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:17.996106    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:43:17.996235 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:17.996115    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:43:17.996235 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.996177    2562 server.go:1268] "Started kubelet"
Sep 16 18:43:17.996235 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.996212    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:43:17.996393 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.996287    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:43:17.996393 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.996366    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:43:17.997964 ip-10-0-128-180 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:43:17.998527 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.998016    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:43:17.999448 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:17.999434    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:43:18.000678 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.000658    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:43:18.002417 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.001335    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-180.ec2.internal.18d5e158b1319e53  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-180.ec2.internal,UID:ip-10-0-128-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-180.ec2.internal,},FirstTimestamp:2026-09-16 18:43:17.995830867 +0000 UTC m=+0.412533616,LastTimestamp:2026-09-16 18:43:17.995830867 +0000 UTC m=+0.412533616,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-180.ec2.internal,}"
Sep 16 18:43:18.004989 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.004971    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:18.005745 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.005731    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:43:18.006075 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.006054    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.006975 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.006953    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:43:18.006975 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.006955    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:43:18.007097 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.006983    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:43:18.007155 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.007107    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:43:18.007155 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.007116    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:43:18.008215 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008190    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:43:18.008215 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008210    2562 factory.go:55] Registering systemd factory
Sep 16 18:43:18.008378 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008220    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:43:18.008685 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008668    2562 factory.go:153] Registering CRI-O factory
Sep 16 18:43:18.008747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008690    2562 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:43:18.008747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008709    2562 factory.go:103] Registering Raw factory
Sep 16 18:43:18.008747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.008725    2562 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:43:18.009393 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.009378    2562 manager.go:349] Starting recovery of all containers
Sep 16 18:43:18.017652 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.017605    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:43:18.017725 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.017637    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:43:18.020600 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.020579    2562 manager.go:354] Recovery completed
Sep 16 18:43:18.024560 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.024547    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.024639 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.024548    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4gprx"
Sep 16 18:43:18.026184 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026167    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.026246 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026197    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.026246 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026209    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.026750 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026735    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:43:18.026750 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026742    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.026750 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026749    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:43:18.026917 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.026767    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:18.027492 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.027459    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.027492 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.027491    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.027561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.027506    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.028896 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.028883    2562 policy_none.go:49] "None policy: Start"
Sep 16 18:43:18.028934 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.028900    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:43:18.028934 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.028910    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:43:18.029891 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.029878    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4gprx"
Sep 16 18:43:18.031102 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.031083    2562 policy_none.go:47] "Start"
Sep 16 18:43:18.034204 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.034143    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-180.ec2.internal.18d5e158b300c749  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-180.ec2.internal,UID:ip-10-0-128-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-180.ec2.internal,},FirstTimestamp:2026-09-16 18:43:18.026184521 +0000 UTC m=+0.442887270,LastTimestamp:2026-09-16 18:43:18.026184521 +0000 UTC m=+0.442887270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-180.ec2.internal,}"
Sep 16 18:43:18.062893 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.062878    2562 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:43:18.063018 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.062914    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:43:18.063018 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.062928    2562 server.go:87] "Starting device plugin registration server"
Sep 16 18:43:18.063192 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.063177    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:43:18.063294 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.063193    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:43:18.063366 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.063338    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:43:18.063416 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.063410    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:43:18.063459 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.063419    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:43:18.064195 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.064177    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:43:18.064269 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.064233    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.115018 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.114850    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:43:18.117211 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.116246    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:43:18.117211 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.116292    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:43:18.117211 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.116313    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:43:18.117211 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.116374    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:43:18.118709 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.118690    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:18.163592 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.163549    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.164638 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.164612    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.165937 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.164763    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.165937 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.164783    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.165937 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.164818    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.171411 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.171396    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.171485 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.171419    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-180.ec2.internal\": node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.197475 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.197447    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.216791 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.216749    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"]
Sep 16 18:43:18.216847 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.216828    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.217615 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.217602    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.217688 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.217627    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.217688 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.217637    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.220034 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220022    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.220182 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220163    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.220245 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220204    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.220773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220757    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.220861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220789    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.220861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220804    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.220861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220814    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.220861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220832    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.220861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.220840    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.223018 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.223004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.223098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.223027    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:18.223834 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.223819    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:18.223931 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.223840    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:18.223931 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.223851    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:18.250040 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.250021    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-180.ec2.internal\" not found" node="ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.254360 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.254345    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-180.ec2.internal\" not found" node="ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.297886 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.297863    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.309420 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.309399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a7f99680109a7b64c9d8a7d4d3d6fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal\" (UID: \"1a7f99680109a7b64c9d8a7d4d3d6fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.309498 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.309430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7f99680109a7b64c9d8a7d4d3d6fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal\" (UID: \"1a7f99680109a7b64c9d8a7d4d3d6fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.309498 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.309449    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73b2ddc0f8f2f0a73b9fa0d48e4b2b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-180.ec2.internal\" (UID: \"73b2ddc0f8f2f0a73b9fa0d48e4b2b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.398404 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.398362    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.409697 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.409676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a7f99680109a7b64c9d8a7d4d3d6fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal\" (UID: \"1a7f99680109a7b64c9d8a7d4d3d6fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.409757 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.409702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7f99680109a7b64c9d8a7d4d3d6fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal\" (UID: \"1a7f99680109a7b64c9d8a7d4d3d6fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.409757 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.409722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73b2ddc0f8f2f0a73b9fa0d48e4b2b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-180.ec2.internal\" (UID: \"73b2ddc0f8f2f0a73b9fa0d48e4b2b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.409856 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.409771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/73b2ddc0f8f2f0a73b9fa0d48e4b2b6e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-180.ec2.internal\" (UID: \"73b2ddc0f8f2f0a73b9fa0d48e4b2b6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.409856 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.409784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a7f99680109a7b64c9d8a7d4d3d6fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal\" (UID: \"1a7f99680109a7b64c9d8a7d4d3d6fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.409856 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.409808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7f99680109a7b64c9d8a7d4d3d6fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal\" (UID: \"1a7f99680109a7b64c9d8a7d4d3d6fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.499149 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.499087    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.552563 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.552535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.557286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.557265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:18.599879 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.599850    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.700362 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.700311    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.800831 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.800755    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.901433 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:18.901401    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:18.933834 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.933811    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:43:18.934218 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:18.933950    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:19.001638 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:19.001609    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:19.033762 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.033729    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:38:18 +0000 UTC" deadline="2028-02-22 06:40:36.79128214 +0000 UTC"
Sep 16 18:43:19.033762 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.033760    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12563h57m17.7575265s"
Sep 16 18:43:19.055945 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.055887    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:19.057378 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.057353    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:19.057486 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.057399    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:19.057486 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.057414    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:19.101685 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:19.101659    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:19.128647 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.128629    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:19.172224 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:19.172190    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b2ddc0f8f2f0a73b9fa0d48e4b2b6e.slice/crio-8b552fb6b58b352932acfccaf5a528a3205bd9b494333f94566c6864dabbf542 WatchSource:0}: Error finding container 8b552fb6b58b352932acfccaf5a528a3205bd9b494333f94566c6864dabbf542: Status 404 returned error can't find the container with id 8b552fb6b58b352932acfccaf5a528a3205bd9b494333f94566c6864dabbf542
Sep 16 18:43:19.172655 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:19.172637    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7f99680109a7b64c9d8a7d4d3d6fef.slice/crio-85f40ef461de2adb601f14282e8a5cf52555217ad07a03abc6568c8f640da2a2 WatchSource:0}: Error finding container 85f40ef461de2adb601f14282e8a5cf52555217ad07a03abc6568c8f640da2a2: Status 404 returned error can't find the container with id 85f40ef461de2adb601f14282e8a5cf52555217ad07a03abc6568c8f640da2a2
Sep 16 18:43:19.179228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.179207    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:19.202600 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:19.202567    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:19.294341 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.294301    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:19.303636 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:19.303603    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-180.ec2.internal\" not found"
Sep 16 18:43:19.386742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.386681    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:19.407439 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.407420    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:19.420405 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.420389    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:19.421288 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.421276    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal"
Sep 16 18:43:19.431222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.431205    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:19.984512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.984477    2562 apiserver.go:52] "Watching apiserver"
Sep 16 18:43:19.991644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.991615    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:43:19.993610 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.993210    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-2thsb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk","openshift-cluster-node-tuning-operator/tuned-fvxlh","openshift-dns/node-resolver-hmdz7","openshift-image-registry/node-ca-4m88x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal","openshift-multus/multus-22sjk","openshift-multus/multus-additional-cni-plugins-mm2fq","kube-system/global-pull-secret-syncer-89j2l","kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal","openshift-multus/network-metrics-daemon-5ps6t","openshift-network-diagnostics/network-check-target-jgp54","openshift-network-operator/iptables-alerter-jqnpc","openshift-ovn-kubernetes/ovnkube-node-lzd4j"]
Sep 16 18:43:19.996359 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.995859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22sjk"
Sep 16 18:43:19.999162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.999131    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:19.999631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.999609    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:43:19.999733 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:19.999696    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:43:20.000318 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.000297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.000428 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.000375    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:43:20.000496 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.000434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w5zp6\""
Sep 16 18:43:20.001917 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.000993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.007032 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.007007    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:20.007523 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.007502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:20.007615 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.007582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.007684 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.007590    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.009167    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.009441    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.009565    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.009857    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010119    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fw9kr\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010179    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9lr5h\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010535    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lcd92\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010672    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.011144 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.010882    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:43:20.012695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.012587    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:43:20.013020 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.013001    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.013941 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.013922    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:20.015484 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.015459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.015585 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.015539    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:20.015849 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.015828    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.016810 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.016646    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:43:20.017982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.017147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:43:20.017982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.017159    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-65lsj\""
Sep 16 18:43:20.017982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.017391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzhlt\""
Sep 16 18:43:20.017982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.017573    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:43:20.017982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.017847    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.018278 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9507241a-c5a0-49a2-89b9-8053a1e2cb34-iptables-alerter-script\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.018278 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:20.018278 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-k8s-cni-cncf-io\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018278 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64cea152-19dd-4c44-9119-5d00570bd11f-multus-daemon-config\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-etc-kubernetes\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-slash\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-var-lib-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-log-socket\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c6d19d5-9dce-406f-a363-e4126893ef82-ovn-node-metrics-cert\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-socket-dir-parent\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-cni-multus\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749bl\" (UniqueName: \"kubernetes.io/projected/64cea152-19dd-4c44-9119-5d00570bd11f-kube-api-access-749bl\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-ovn\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/572231be-7380-44a9-8379-caf0f42c820d-serviceca\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-multus-certs\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-node-log\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9507241a-c5a0-49a2-89b9-8053a1e2cb34-host-slash\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-tmp-dir\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-cni-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-ovnkube-script-lib\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-hosts-file\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lsbm\" (UniqueName: \"kubernetes.io/projected/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-kube-api-access-9lsbm\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-cnibin\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-kubelet\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018872    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.018899 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-kubelet\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-cni-netd\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.018945    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.018972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/572231be-7380-44a9-8379-caf0f42c820d-host\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-system-cni-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-netns\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-ovnkube-config\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-env-overrides\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9pb\" (UniqueName: \"kubernetes.io/projected/8c6d19d5-9dce-406f-a363-e4126893ef82-kube-api-access-4h9pb\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzncx\" (UniqueName: \"kubernetes.io/projected/572231be-7380-44a9-8379-caf0f42c820d-kube-api-access-dzncx\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-cni-bin\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-hostroot\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-conf-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-systemd-units\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.019633 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-etc-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.020372 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-cni-bin\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.020372 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9v4\" (UniqueName: \"kubernetes.io/projected/9507241a-c5a0-49a2-89b9-8053a1e2cb34-kube-api-access-4m9v4\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.020372 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64cea152-19dd-4c44-9119-5d00570bd11f-cni-binary-copy\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.020372 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-run-netns\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.020372 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.019563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-systemd\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.021732 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.021159    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.021833 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.021809    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:43:20.022655 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.022634    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.023664 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.023645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.024328 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.024306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:43:20.026190 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.026171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.026713 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.026694    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:43:20.026822 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.026774    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8565d\""
Sep 16 18:43:20.030927 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.030907    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.032186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.032166    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:43:20.032446 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.032170    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:43:20.032640 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.032617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5vqlh\""
Sep 16 18:43:20.032953 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.032935    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:20.033159 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.033140    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qr82x\""
Sep 16 18:43:20.033303 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.033288    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:43:20.053956 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.053662    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:20.088935 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.088914    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7c5r"
Sep 16 18:43:20.108961 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.108904    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:43:20.109948 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.109912    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7c5r"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-system-cni-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-netns\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-env-overrides\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9pb\" (UniqueName: \"kubernetes.io/projected/8c6d19d5-9dce-406f-a363-e4126893ef82-kube-api-access-4h9pb\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysctl-d\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-host\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-hostroot\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-conf-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-etc-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-cni-bin\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-system-cni-dir\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.119990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwsp\" (UniqueName: \"kubernetes.io/projected/9f3702ff-8beb-4aab-8185-92870a17d93d-kube-api-access-rmwsp\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-run-netns\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120041    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-var-lib-kubelet\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.126092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-socket-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-registration-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-k8s-cni-cncf-io\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64cea152-19dd-4c44-9119-5d00570bd11f-multus-daemon-config\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120193    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-etc-kubernetes\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-var-lib-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c6d19d5-9dce-406f-a363-e4126893ef82-ovn-node-metrics-cert\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-cni-multus\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749bl\" (UniqueName: \"kubernetes.io/projected/64cea152-19dd-4c44-9119-5d00570bd11f-kube-api-access-749bl\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-ovn\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-systemd\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-sys\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-lib-modules\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-cnibin\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/572231be-7380-44a9-8379-caf0f42c820d-serviceca\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-multus-certs\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-node-log\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9507241a-c5a0-49a2-89b9-8053a1e2cb34-host-slash\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-dbus\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-modprobe-d\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnd4\" (UniqueName: \"kubernetes.io/projected/2cb5a23e-f4e8-4542-b175-5fe7be200604-kube-api-access-ctnd4\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-ovnkube-script-lib\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-hosts-file\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-device-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-cnibin\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-cni-netd\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4d6f9b7-eb42-43f6-a88e-70a0f332226b-agent-certs\") pod \"konnectivity-agent-2thsb\" (UID: \"b4d6f9b7-eb42-43f6-a88e-70a0f332226b\") " pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-kubernetes\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-run\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4d6f9b7-eb42-43f6-a88e-70a0f332226b-konnectivity-ca\") pod \"konnectivity-agent-2thsb\" (UID: \"b4d6f9b7-eb42-43f6-a88e-70a0f332226b\") " pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.120984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-ovnkube-config\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.127901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysctl-conf\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-etc-selinux\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzncx\" (UniqueName: \"kubernetes.io/projected/572231be-7380-44a9-8379-caf0f42c820d-kube-api-access-dzncx\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-cni-bin\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-systemd-units\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9v4\" (UniqueName: \"kubernetes.io/projected/9507241a-c5a0-49a2-89b9-8053a1e2cb34-kube-api-access-4m9v4\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8577edcc-7d4d-46b4-bcba-06e522ede20b-tmp\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64cea152-19dd-4c44-9119-5d00570bd11f-cni-binary-copy\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-systemd\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9507241a-c5a0-49a2-89b9-8053a1e2cb34-iptables-alerter-script\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysconfig\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-slash\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-log-socket\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-sys-fs\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-socket-dir-parent\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.128710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-tmp-dir\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzsjt\" (UniqueName: \"kubernetes.io/projected/3c9ced61-ba03-421f-bda1-20977b70864c-kube-api-access-dzsjt\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-cni-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lsbm\" (UniqueName: \"kubernetes.io/projected/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-kube-api-access-9lsbm\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-kubelet-config\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6r8\" (UniqueName: \"kubernetes.io/projected/8577edcc-7d4d-46b4-bcba-06e522ede20b-kube-api-access-dm6r8\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-kubelet\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-kubelet\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-tuned\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/572231be-7380-44a9-8379-caf0f42c820d-host\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/572231be-7380-44a9-8379-caf0f42c820d-host\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-system-cni-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.121905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-netns\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.129603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.122372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-env-overrides\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.122680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-hostroot\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.122720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-conf-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.123200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-etc-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.123245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-cni-bin\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.123305    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-run-netns\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.123385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-k8s-cni-cncf-io\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.123933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64cea152-19dd-4c44-9119-5d00570bd11f-multus-daemon-config\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-socket-dir-parent\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124198    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-systemd\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124212    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-tmp-dir\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-cni-bin\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9507241a-c5a0-49a2-89b9-8053a1e2cb34-iptables-alerter-script\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-multus-cni-dir\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-systemd-units\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-slash\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.124961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-log-socket\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.130473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-kubelet\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64cea152-19dd-4c44-9119-5d00570bd11f-cni-binary-copy\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125097    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9507241a-c5a0-49a2-89b9-8053a1e2cb34-host-slash\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125115    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-kubelet\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-etc-kubernetes\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-cni-netd\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-cnibin\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-var-lib-openvswitch\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-var-lib-cni-multus\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-run-ovn\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-host-run-ovn-kubernetes\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-hosts-file\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64cea152-19dd-4c44-9119-5d00570bd11f-host-run-multus-certs\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c6d19d5-9dce-406f-a363-e4126893ef82-node-log\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125702    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.125842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/572231be-7380-44a9-8379-caf0f42c820d-serviceca\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.126007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal" event={"ID":"73b2ddc0f8f2f0a73b9fa0d48e4b2b6e","Type":"ContainerStarted","Data":"8b552fb6b58b352932acfccaf5a528a3205bd9b494333f94566c6864dabbf542"}
Sep 16 18:43:20.131412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.126031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-ovnkube-script-lib\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.132316 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.126851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c6d19d5-9dce-406f-a363-e4126893ef82-ovnkube-config\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.132316 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.128279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal" event={"ID":"1a7f99680109a7b64c9d8a7d4d3d6fef","Type":"ContainerStarted","Data":"85f40ef461de2adb601f14282e8a5cf52555217ad07a03abc6568c8f640da2a2"}
Sep 16 18:43:20.132316 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.130027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c6d19d5-9dce-406f-a363-e4126893ef82-ovn-node-metrics-cert\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.135315 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.135257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9pb\" (UniqueName: \"kubernetes.io/projected/8c6d19d5-9dce-406f-a363-e4126893ef82-kube-api-access-4h9pb\") pod \"ovnkube-node-lzd4j\" (UID: \"8c6d19d5-9dce-406f-a363-e4126893ef82\") " pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.136185 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.135956    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:20.136185 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.135980    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:20.136185 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.135995    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:20.136185 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.136047    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:20.636031872 +0000 UTC m=+3.052734611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:20.141164 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.138146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzncx\" (UniqueName: \"kubernetes.io/projected/572231be-7380-44a9-8379-caf0f42c820d-kube-api-access-dzncx\") pod \"node-ca-4m88x\" (UID: \"572231be-7380-44a9-8379-caf0f42c820d\") " pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.141164 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.138577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9v4\" (UniqueName: \"kubernetes.io/projected/9507241a-c5a0-49a2-89b9-8053a1e2cb34-kube-api-access-4m9v4\") pod \"iptables-alerter-jqnpc\" (UID: \"9507241a-c5a0-49a2-89b9-8053a1e2cb34\") " pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.141164 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.140234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-749bl\" (UniqueName: \"kubernetes.io/projected/64cea152-19dd-4c44-9119-5d00570bd11f-kube-api-access-749bl\") pod \"multus-22sjk\" (UID: \"64cea152-19dd-4c44-9119-5d00570bd11f\") " pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.143183 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.143157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lsbm\" (UniqueName: \"kubernetes.io/projected/3f93bc5c-ba37-4cbc-8c8e-9062254a738c-kube-api-access-9lsbm\") pod \"node-resolver-hmdz7\" (UID: \"3f93bc5c-ba37-4cbc-8c8e-9062254a738c\") " pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.222974 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.222946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-sys-fs\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.222974 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.222978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.222996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzsjt\" (UniqueName: \"kubernetes.io/projected/3c9ced61-ba03-421f-bda1-20977b70864c-kube-api-access-dzsjt\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-kubelet-config\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6r8\" (UniqueName: \"kubernetes.io/projected/8577edcc-7d4d-46b4-bcba-06e522ede20b-kube-api-access-dm6r8\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-tuned\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-sys-fs\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.223120    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysctl-d\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223212 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.223199    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:20.723176418 +0000 UTC m=+3.139879180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-host\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-system-cni-dir\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysctl-d\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwsp\" (UniqueName: \"kubernetes.io/projected/9f3702ff-8beb-4aab-8185-92870a17d93d-kube-api-access-rmwsp\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-host\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-var-lib-kubelet\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-system-cni-dir\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-socket-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-registration-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-systemd\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-sys\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-lib-modules\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-cnibin\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.223773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223622    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c9ced61-ba03-421f-bda1-20977b70864c-cnibin\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-socket-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-var-lib-kubelet\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-registration-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224194    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-systemd\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-sys\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.223623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c9ced61-ba03-421f-bda1-20977b70864c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-dbus\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-modprobe-d\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-kubelet-config\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnd4\" (UniqueName: \"kubernetes.io/projected/2cb5a23e-f4e8-4542-b175-5fe7be200604-kube-api-access-ctnd4\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-device-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224518    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-dbus\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.224727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4d6f9b7-eb42-43f6-a88e-70a0f332226b-agent-certs\") pod \"konnectivity-agent-2thsb\" (UID: \"b4d6f9b7-eb42-43f6-a88e-70a0f332226b\") " pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-kubernetes\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-run\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4d6f9b7-eb42-43f6-a88e-70a0f332226b-konnectivity-ca\") pod \"konnectivity-agent-2thsb\" (UID: \"b4d6f9b7-eb42-43f6-a88e-70a0f332226b\") " pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-modprobe-d\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysctl-conf\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-etc-selinux\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-lib-modules\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8577edcc-7d4d-46b4-bcba-06e522ede20b-tmp\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224707    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-run\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-kubernetes\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysconfig\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-device-dir\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysconfig\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.224875    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.224928    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:43:20.724910623 +0000 UTC m=+3.141613375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:20.225590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.224941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9f3702ff-8beb-4aab-8185-92870a17d93d-etc-selinux\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.226180 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.225047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-sysctl-conf\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.226180 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.225327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4d6f9b7-eb42-43f6-a88e-70a0f332226b-konnectivity-ca\") pod \"konnectivity-agent-2thsb\" (UID: \"b4d6f9b7-eb42-43f6-a88e-70a0f332226b\") " pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.227085 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.227055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8577edcc-7d4d-46b4-bcba-06e522ede20b-etc-tuned\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.227454 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.227433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4d6f9b7-eb42-43f6-a88e-70a0f332226b-agent-certs\") pod \"konnectivity-agent-2thsb\" (UID: \"b4d6f9b7-eb42-43f6-a88e-70a0f332226b\") " pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.227634 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.227613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8577edcc-7d4d-46b4-bcba-06e522ede20b-tmp\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.239060 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.239039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6r8\" (UniqueName: \"kubernetes.io/projected/8577edcc-7d4d-46b4-bcba-06e522ede20b-kube-api-access-dm6r8\") pod \"tuned-fvxlh\" (UID: \"8577edcc-7d4d-46b4-bcba-06e522ede20b\") " pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.239343 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.239313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwsp\" (UniqueName: \"kubernetes.io/projected/9f3702ff-8beb-4aab-8185-92870a17d93d-kube-api-access-rmwsp\") pod \"aws-ebs-csi-driver-node-k72rk\" (UID: \"9f3702ff-8beb-4aab-8185-92870a17d93d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.240308 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.240286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnd4\" (UniqueName: \"kubernetes.io/projected/2cb5a23e-f4e8-4542-b175-5fe7be200604-kube-api-access-ctnd4\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.242864 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.242841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzsjt\" (UniqueName: \"kubernetes.io/projected/3c9ced61-ba03-421f-bda1-20977b70864c-kube-api-access-dzsjt\") pod \"multus-additional-cni-plugins-mm2fq\" (UID: \"3c9ced61-ba03-421f-bda1-20977b70864c\") " pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.311644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.311610    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:20.323090 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.323064    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22sjk"
Sep 16 18:43:20.332541 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.332517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:20.342755 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.342735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqnpc"
Sep 16 18:43:20.349644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.349620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmdz7"
Sep 16 18:43:20.357554 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.357535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4m88x"
Sep 16 18:43:20.367387 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.367354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mm2fq"
Sep 16 18:43:20.374508 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.374491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:20.383457 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.383431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fvxlh"
Sep 16 18:43:20.389508 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.389488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk"
Sep 16 18:43:20.729861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.729767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:20.729861 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.729824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:20.729863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.729954    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.729977    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.729987    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.729992    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.730008    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:20.730056 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.730052    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:43:21.730032572 +0000 UTC m=+4.146735326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:20.730273 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.730075    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:21.730057947 +0000 UTC m=+4.146760691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:20.730273 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:20.730098    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:21.730085974 +0000 UTC m=+4.146788713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:21.114162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:21.114053    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:20 +0000 UTC" deadline="2028-02-28 18:14:25.741769229 +0000 UTC"
Sep 16 18:43:21.114162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:21.114096    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12719h31m4.627678602s"
Sep 16 18:43:21.285448 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.285415    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572231be_7380_44a9_8379_caf0f42c820d.slice/crio-e0c8e3e6cd8c4bf58c38dcfc8fb85d3d0b379951ad975991f1ac4cf31e0740f9 WatchSource:0}: Error finding container e0c8e3e6cd8c4bf58c38dcfc8fb85d3d0b379951ad975991f1ac4cf31e0740f9: Status 404 returned error can't find the container with id e0c8e3e6cd8c4bf58c38dcfc8fb85d3d0b379951ad975991f1ac4cf31e0740f9
Sep 16 18:43:21.286509 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.286480    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cea152_19dd_4c44_9119_5d00570bd11f.slice/crio-d1055363b152cab0ab0b264e89e4d220590cf6b7e0ba50f331ad3ae9df85cd0b WatchSource:0}: Error finding container d1055363b152cab0ab0b264e89e4d220590cf6b7e0ba50f331ad3ae9df85cd0b: Status 404 returned error can't find the container with id d1055363b152cab0ab0b264e89e4d220590cf6b7e0ba50f331ad3ae9df85cd0b
Sep 16 18:43:21.288919 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.288724    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3702ff_8beb_4aab_8185_92870a17d93d.slice/crio-bdf00fcd45051274e0e75ab3572f93e04fb27d7dea853c09e4ca398fe412120a WatchSource:0}: Error finding container bdf00fcd45051274e0e75ab3572f93e04fb27d7dea853c09e4ca398fe412120a: Status 404 returned error can't find the container with id bdf00fcd45051274e0e75ab3572f93e04fb27d7dea853c09e4ca398fe412120a
Sep 16 18:43:21.292306 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.292271    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9507241a_c5a0_49a2_89b9_8053a1e2cb34.slice/crio-1ed1c29d9bcffb1cb4ff5b1106bbc0df71a7a2b9e6b5d2f26a91b3473c6b1ab2 WatchSource:0}: Error finding container 1ed1c29d9bcffb1cb4ff5b1106bbc0df71a7a2b9e6b5d2f26a91b3473c6b1ab2: Status 404 returned error can't find the container with id 1ed1c29d9bcffb1cb4ff5b1106bbc0df71a7a2b9e6b5d2f26a91b3473c6b1ab2
Sep 16 18:43:21.293452 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.293421    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f93bc5c_ba37_4cbc_8c8e_9062254a738c.slice/crio-c63ff23a818154463cde4f3a5ace531ee982eecec1b5ff4ef9cb8309c232ad95 WatchSource:0}: Error finding container c63ff23a818154463cde4f3a5ace531ee982eecec1b5ff4ef9cb8309c232ad95: Status 404 returned error can't find the container with id c63ff23a818154463cde4f3a5ace531ee982eecec1b5ff4ef9cb8309c232ad95
Sep 16 18:43:21.294491 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.294464    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8577edcc_7d4d_46b4_bcba_06e522ede20b.slice/crio-0ba3093720f15a0100869a6ee5fdf3479e27a230f549fea3a9af34f09b61ae59 WatchSource:0}: Error finding container 0ba3093720f15a0100869a6ee5fdf3479e27a230f549fea3a9af34f09b61ae59: Status 404 returned error can't find the container with id 0ba3093720f15a0100869a6ee5fdf3479e27a230f549fea3a9af34f09b61ae59
Sep 16 18:43:21.295064 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.295026    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9ced61_ba03_421f_bda1_20977b70864c.slice/crio-e61eafe632fb0fabe548b8ac0406df5143bd315539379f78a8f12a4bf60172f1 WatchSource:0}: Error finding container e61eafe632fb0fabe548b8ac0406df5143bd315539379f78a8f12a4bf60172f1: Status 404 returned error can't find the container with id e61eafe632fb0fabe548b8ac0406df5143bd315539379f78a8f12a4bf60172f1
Sep 16 18:43:21.296419 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:21.296211    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d6f9b7_eb42_43f6_a88e_70a0f332226b.slice/crio-465e5fb832b4259e9c6b1b022f368a410126a59729e41e379aab3a13e46ad011 WatchSource:0}: Error finding container 465e5fb832b4259e9c6b1b022f368a410126a59729e41e379aab3a13e46ad011: Status 404 returned error can't find the container with id 465e5fb832b4259e9c6b1b022f368a410126a59729e41e379aab3a13e46ad011
Sep 16 18:43:21.737045 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:21.736766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:21.737045 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.736974    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:21.737045 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.736999    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:21.737045 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:21.736999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:21.737045 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.737009    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:21.737045 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:21.737028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:21.737484 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.737059    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:23.737045295 +0000 UTC m=+6.153748048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:21.737484 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.737130    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:21.737484 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.737178    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:23.737164332 +0000 UTC m=+6.153867071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:21.737484 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.737240    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:21.737484 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:21.737280    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:43:23.737265604 +0000 UTC m=+6.153968375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:22.114490 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.114303    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:20 +0000 UTC" deadline="2028-04-12 16:27:55.83558403 +0000 UTC"
Sep 16 18:43:22.114490 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.114391    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13773h44m33.721197473s"
Sep 16 18:43:22.119315 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.119291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:22.119490 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:22.119435    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:22.119878 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.119857    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:22.119986 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:22.119964    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:22.120050 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.120038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:22.120149 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:22.120125    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:22.144721 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.144650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2thsb" event={"ID":"b4d6f9b7-eb42-43f6-a88e-70a0f332226b","Type":"ContainerStarted","Data":"465e5fb832b4259e9c6b1b022f368a410126a59729e41e379aab3a13e46ad011"}
Sep 16 18:43:22.150554 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.150470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerStarted","Data":"e61eafe632fb0fabe548b8ac0406df5143bd315539379f78a8f12a4bf60172f1"}
Sep 16 18:43:22.168686 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.168561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fvxlh" event={"ID":"8577edcc-7d4d-46b4-bcba-06e522ede20b","Type":"ContainerStarted","Data":"0ba3093720f15a0100869a6ee5fdf3479e27a230f549fea3a9af34f09b61ae59"}
Sep 16 18:43:22.178005 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.177917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqnpc" event={"ID":"9507241a-c5a0-49a2-89b9-8053a1e2cb34","Type":"ContainerStarted","Data":"1ed1c29d9bcffb1cb4ff5b1106bbc0df71a7a2b9e6b5d2f26a91b3473c6b1ab2"}
Sep 16 18:43:22.183756 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.183675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22sjk" event={"ID":"64cea152-19dd-4c44-9119-5d00570bd11f","Type":"ContainerStarted","Data":"d1055363b152cab0ab0b264e89e4d220590cf6b7e0ba50f331ad3ae9df85cd0b"}
Sep 16 18:43:22.191830 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.191747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4m88x" event={"ID":"572231be-7380-44a9-8379-caf0f42c820d","Type":"ContainerStarted","Data":"e0c8e3e6cd8c4bf58c38dcfc8fb85d3d0b379951ad975991f1ac4cf31e0740f9"}
Sep 16 18:43:22.204285 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.203546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal" event={"ID":"73b2ddc0f8f2f0a73b9fa0d48e4b2b6e","Type":"ContainerStarted","Data":"5a7bed9c3be2fc4c49f5c9b8b9d844aa2ad4bd584480887a4108e7c4d3388f12"}
Sep 16 18:43:22.209086 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.208449    2562 generic.go:356] "Generic (PLEG): container finished" podID="1a7f99680109a7b64c9d8a7d4d3d6fef" containerID="83721d3ebdc6f85621fb13346d064a718e337b434175b6e59eb2742658bfcda4" exitCode=0
Sep 16 18:43:22.209086 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.208535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal" event={"ID":"1a7f99680109a7b64c9d8a7d4d3d6fef","Type":"ContainerDied","Data":"83721d3ebdc6f85621fb13346d064a718e337b434175b6e59eb2742658bfcda4"}
Sep 16 18:43:22.213297 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.213269    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"553a48a90742f939182219d8af2370711d8e685da14f407112dca6aa85ac65b5"}
Sep 16 18:43:22.221104 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.220911    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-180.ec2.internal" podStartSLOduration=2.220893101 podStartE2EDuration="2.220893101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:22.219742334 +0000 UTC m=+4.636445098" watchObservedRunningTime="2026-09-16 18:43:22.220893101 +0000 UTC m=+4.637595865"
Sep 16 18:43:22.221533 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.221485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmdz7" event={"ID":"3f93bc5c-ba37-4cbc-8c8e-9062254a738c","Type":"ContainerStarted","Data":"c63ff23a818154463cde4f3a5ace531ee982eecec1b5ff4ef9cb8309c232ad95"}
Sep 16 18:43:22.225167 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:22.225136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk" event={"ID":"9f3702ff-8beb-4aab-8185-92870a17d93d","Type":"ContainerStarted","Data":"bdf00fcd45051274e0e75ab3572f93e04fb27d7dea853c09e4ca398fe412120a"}
Sep 16 18:43:23.244527 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:23.244490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal" event={"ID":"1a7f99680109a7b64c9d8a7d4d3d6fef","Type":"ContainerStarted","Data":"9705584ce887fa780fa5e295735b90b5baff059ff7c85c45e1985074c71b2acf"}
Sep 16 18:43:23.262347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:23.261427    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-180.ec2.internal" podStartSLOduration=3.261388801 podStartE2EDuration="3.261388801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:23.261194843 +0000 UTC m=+5.677897624" watchObservedRunningTime="2026-09-16 18:43:23.261388801 +0000 UTC m=+5.678091565"
Sep 16 18:43:23.755287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:23.755242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:23.755566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:23.755304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:23.755566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:23.755339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:23.755566 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.755497    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:23.755566 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.755564    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:27.755544109 +0000 UTC m=+10.172246867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:23.756077 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.756011    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:23.756077 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.756036    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:23.756077 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.756049    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:23.756274 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.756101    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:27.756083204 +0000 UTC m=+10.172785943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:23.756274 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.756175    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:23.756274 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:23.756210    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:43:27.756199745 +0000 UTC m=+10.172902485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:24.118258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:24.117468    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:24.118258 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:24.117572    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:24.118258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:24.117924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:24.118258 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:24.118021    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:24.118258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:24.118084    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:24.118258 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:24.118160    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:26.120099 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:26.119522    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:26.120099 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:26.119674    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:26.120099 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:26.120002    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:26.120099 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:26.120096    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:26.120778 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:26.120142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:26.120778 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:26.120212    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:27.791067 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:27.790974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:27.791067 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:27.791026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:27.791429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791544    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791594    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791613    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791624    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791629    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:43:35.79159647 +0000 UTC m=+18.208299227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:27.791688 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791677    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:35.79166456 +0000 UTC m=+18.208367299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:27.792047 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791752    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:27.792047 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:27.791795    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:35.79178489 +0000 UTC m=+18.208487629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:28.121101 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:28.120538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:28.121101 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:28.120545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:28.121101 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:28.120681    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:28.121101 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:28.120685    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:28.121101 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:28.120754    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:28.121101 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:28.120828    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:30.117299 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:30.117263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:30.117789 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:30.117310    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:30.117789 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:30.117263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:30.117789 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:30.117428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:30.117789 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:30.117536    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:30.117789 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:30.117628    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:32.116595 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:32.116555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:32.117025 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:32.116612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:32.117025 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:32.116559    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:32.117025 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:32.116685    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:32.117025 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:32.116806    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:32.117025 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:32.116909    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:34.117286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:34.117250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:34.117650 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:34.117391    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:34.118098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:34.117786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:34.118098 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:34.117883    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:34.118098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:34.117971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:34.118098 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:34.118065    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:35.856407 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:35.856370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:35.856886 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:35.856415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:35.856886 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:35.856434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:35.856886 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.856569    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:35.856886 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.856624    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.856610204 +0000 UTC m=+34.273312941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:35.857039 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.856997    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:35.857039 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.857013    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:35.857039 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.857027    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:35.857135 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.857064    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.857051202 +0000 UTC m=+34.273753940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:35.857135 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.857117    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:35.857213 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:35.857140    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:43:51.857132707 +0000 UTC m=+34.273835444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:36.117109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:36.117024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:36.117109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:36.117031    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:36.117109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:36.117069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:36.117395 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:36.117173    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:36.117395 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:36.117303    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:36.121498 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:36.121264    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:38.118768 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:38.118297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:38.118768 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:38.118406    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:38.118768 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:38.118423    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:38.118768 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:38.118538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:38.118768 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:38.118544    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:38.118768 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:38.118616    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:40.116737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:40.116696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:40.116737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:40.116734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:40.117230 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:40.116703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:40.117230 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:40.116834    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:40.117230 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:40.116907    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:40.117230 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:40.116985    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:41.286286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.285870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2thsb" event={"ID":"b4d6f9b7-eb42-43f6-a88e-70a0f332226b","Type":"ContainerStarted","Data":"a8ca2d6c84967d972ef9d677f78767cb8905202ec34bc0a829d0ead2bfb07475"}
Sep 16 18:43:41.287558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.287529    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c9ced61-ba03-421f-bda1-20977b70864c" containerID="ac57a585611726b5607c5a1538453cddca84ab5d76e278947028ce9a97de7a53" exitCode=0
Sep 16 18:43:41.289956 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.287612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerDied","Data":"ac57a585611726b5607c5a1538453cddca84ab5d76e278947028ce9a97de7a53"}
Sep 16 18:43:41.291941 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.291915    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fvxlh" event={"ID":"8577edcc-7d4d-46b4-bcba-06e522ede20b","Type":"ContainerStarted","Data":"93a3ce8c5349b2f27ce50c4a8efacc6d164f2b0437ac1cb625733adbd5ef87d3"}
Sep 16 18:43:41.293577 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.293555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22sjk" event={"ID":"64cea152-19dd-4c44-9119-5d00570bd11f","Type":"ContainerStarted","Data":"315c999745715de6cb5850c37e05781af10133672d2cb5cb79d0e07a98074ea1"}
Sep 16 18:43:41.295018 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.294998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4m88x" event={"ID":"572231be-7380-44a9-8379-caf0f42c820d","Type":"ContainerStarted","Data":"12a3ad4676cf852bd7770b6083451cd3279c45c0dbdd4226c4036995ea363d5f"}
Sep 16 18:43:41.297874 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.297851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"6079a0fa008e3cb8a3473852ce7357e8c991ba5d76ec3b1b9735f866343c0116"}
Sep 16 18:43:41.297964 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.297924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"2f28995ee0cdb94bd9c8ca0a37875455d315f9ea36fc03d013a5cdf59911286b"}
Sep 16 18:43:41.298008 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.297976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"7bf0505a625a8efb92ca5746d8a61e715f29a8e87a3cf6852a39d737c9006c07"}
Sep 16 18:43:41.298062 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.298047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"a64f48210a2895602551ef7283de156809e3027860e229fda215727c558bf34f"}
Sep 16 18:43:41.298106 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.298095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"1c6b8f5eaabbdc302eb1d1cce2994455571128f35919d48249d7586ece6627c9"}
Sep 16 18:43:41.298151 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.298135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"b4c3bc678ed50ced49bc4a8fb3bc6e92bd6aea492014a216da6666e04d7befd0"}
Sep 16 18:43:41.299291 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.299273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmdz7" event={"ID":"3f93bc5c-ba37-4cbc-8c8e-9062254a738c","Type":"ContainerStarted","Data":"b9ea07ca1e79f6ea80bc9f69abfabb03c1b20c0e760dfb4c630c321c46f6e3f6"}
Sep 16 18:43:41.300674 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.300656    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk" event={"ID":"9f3702ff-8beb-4aab-8185-92870a17d93d","Type":"ContainerStarted","Data":"f1a9961d92de005315fe8d248cfad99d0f4d266f42f03fa0d81d123eb91cb7ca"}
Sep 16 18:43:41.303483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.303450    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2thsb" podStartSLOduration=7.160201279 podStartE2EDuration="22.303438758s" totalImagesPullingTime="15.143237479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.302988005 +0000 UTC m=+23.719690764" watchObservedRunningTime="2026-09-16 18:43:41.303438758 +0000 UTC m=+23.720141517"
Sep 16 18:43:41.387787 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.387745    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fvxlh" podStartSLOduration=3.5173501270000003 podStartE2EDuration="22.38773122s" totalImagesPullingTime="18.870381093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.360854738 +0000 UTC m=+23.777557495" watchObservedRunningTime="2026-09-16 18:43:41.38773122 +0000 UTC m=+23.804433978"
Sep 16 18:43:41.387924 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.387906    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-22sjk" podStartSLOduration=4.472404344 podStartE2EDuration="23.387901243s" totalImagesPullingTime="18.915496899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.387644591 +0000 UTC m=+23.804347349" watchObservedRunningTime="2026-09-16 18:43:41.387901243 +0000 UTC m=+23.804604001"
Sep 16 18:43:41.410157 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.410107    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmdz7" podStartSLOduration=3.555798673 podStartE2EDuration="22.410090323s" totalImagesPullingTime="18.85429165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.409726887 +0000 UTC m=+23.826429646" watchObservedRunningTime="2026-09-16 18:43:41.410090323 +0000 UTC m=+23.826793083"
Sep 16 18:43:41.445472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.445434    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4m88x" podStartSLOduration=3.582669947 podStartE2EDuration="22.445416936s" totalImagesPullingTime="18.862746989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.445314027 +0000 UTC m=+23.862016786" watchObservedRunningTime="2026-09-16 18:43:41.445416936 +0000 UTC m=+23.862119693"
Sep 16 18:43:41.629463 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:41.629430    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:43:42.076391 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.076148    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:43:41.629451827Z","UUID":"be15ce16-ea54-461e-9f4d-6c37e5747784","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:43:42.077972 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.077945    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:43:42.078118 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.077986    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:43:42.117177 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.117147    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:42.117374 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:42.117293    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:42.117718 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.117689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:42.117820 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:42.117786    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:42.117874 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.117858    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:42.117947 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:42.117930    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:42.306657 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.306621    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk" event={"ID":"9f3702ff-8beb-4aab-8185-92870a17d93d","Type":"ContainerStarted","Data":"fc9f3944913c0e220ff447e8a6984cb65fd3539f9aa0634931d2bb931fedaa73"}
Sep 16 18:43:42.309506 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.309460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqnpc" event={"ID":"9507241a-c5a0-49a2-89b9-8053a1e2cb34","Type":"ContainerStarted","Data":"b336505824199a507d676a36cfadb7d26cce9bf3678245336f52044d928078ff"}
Sep 16 18:43:42.326064 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:42.326007    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jqnpc" podStartSLOduration=5.470405866 podStartE2EDuration="24.325992299s" totalImagesPullingTime="18.855586433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:42.32566832 +0000 UTC m=+24.742371099" watchObservedRunningTime="2026-09-16 18:43:42.325992299 +0000 UTC m=+24.742695057"
Sep 16 18:43:43.314681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:43.314635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"3c2ff58a2a2acde728c7beddf2a74850c030f1b007fa865b574581475250bdf0"}
Sep 16 18:43:43.317076 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:43.317045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk" event={"ID":"9f3702ff-8beb-4aab-8185-92870a17d93d","Type":"ContainerStarted","Data":"995bb7524bdce60e7cea2a263bb6af3f70920f23a8dd4520ee59629fd0ff46d7"}
Sep 16 18:43:43.340911 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:43.340867    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k72rk" podStartSLOduration=3.455605081 podStartE2EDuration="24.340854071s" totalImagesPullingTime="20.88524899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:43.340755805 +0000 UTC m=+25.757458577" watchObservedRunningTime="2026-09-16 18:43:43.340854071 +0000 UTC m=+25.757556830"
Sep 16 18:43:44.116541 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:44.116506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:44.116541 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:44.116525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:44.116776 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:44.116507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:44.116776 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:44.116634    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:44.116776 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:44.116703    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:44.116947 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:44.116774    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:46.116974 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.116795    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:46.117658 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.116794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:46.117658 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:46.117041    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:46.117658 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:46.117119    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:46.117658 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.116797    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:46.117658 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:46.117255    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:46.250232 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.250199    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:46.250928 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.250911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:46.325355 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.325308    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c9ced61-ba03-421f-bda1-20977b70864c" containerID="1e93a31cab02d2a482d2be4aa092c76275ad9471e64f4b5dc8728ca9bd0382ec" exitCode=0
Sep 16 18:43:46.325478 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.325391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerDied","Data":"1e93a31cab02d2a482d2be4aa092c76275ad9471e64f4b5dc8728ca9bd0382ec"}
Sep 16 18:43:46.329490 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.329458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" event={"ID":"8c6d19d5-9dce-406f-a363-e4126893ef82","Type":"ContainerStarted","Data":"ca5f70aacfc293a9591e25723e89aaef14119cca90b5f8d88608fadc508c02f1"}
Sep 16 18:43:46.329598 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.329580    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:46.330084 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.330065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2thsb"
Sep 16 18:43:46.381512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:46.381409    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j" podStartSLOduration=9.479088234 podStartE2EDuration="28.381394022s" totalImagesPullingTime="18.902305788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:46.380866574 +0000 UTC m=+28.797569346" watchObservedRunningTime="2026-09-16 18:43:46.381394022 +0000 UTC m=+28.798096781"
Sep 16 18:43:47.332089 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.331915    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:47.332548 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.332124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:47.332548 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.332157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:47.347791 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.347769    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:47.348336 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.348297    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:43:47.356241 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.356219    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89j2l"]
Sep 16 18:43:47.356359 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.356314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:47.356479 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:47.356457    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:47.356953 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.356933    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ps6t"]
Sep 16 18:43:47.357049 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.357025    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:47.357148 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:47.357131    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:47.360572 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.360547    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jgp54"]
Sep 16 18:43:47.360769 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:47.360755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:47.361075 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:47.361023    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:48.335555 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:48.335522    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c9ced61-ba03-421f-bda1-20977b70864c" containerID="a9c2087559a9a531f31569c5d54258ac2acf41f2b0808b4e22dfb7850dffb006" exitCode=0
Sep 16 18:43:48.335963 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:48.335599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerDied","Data":"a9c2087559a9a531f31569c5d54258ac2acf41f2b0808b4e22dfb7850dffb006"}
Sep 16 18:43:49.117501 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:49.117471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:49.117674 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:49.117472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:49.117674 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:49.117596    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:49.117674 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:49.117486    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:49.117824 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:49.117718    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:49.117824 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:49.117794    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:50.342532 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:50.342498    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c9ced61-ba03-421f-bda1-20977b70864c" containerID="4e840e670c3018f975610d143d5d89e22afd9f1ec05ebf8bda5d5361220d5146" exitCode=0
Sep 16 18:43:50.342969 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:50.342557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerDied","Data":"4e840e670c3018f975610d143d5d89e22afd9f1ec05ebf8bda5d5361220d5146"}
Sep 16 18:43:51.117089 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:51.117051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:51.117297 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:51.117179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:51.117297 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.117187    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:51.117297 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:51.117215    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:51.117464 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.117338    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:51.117504 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.117457    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:51.872896 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:51.872857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:51.872909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873018    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873042    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873067    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873077    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret podName:4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab nodeName:}" failed. No retries permitted until 2026-09-16 18:44:23.873059364 +0000 UTC m=+66.289762125 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret") pod "global-pull-secret-syncer-89j2l" (UID: "4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873081    2562 projected.go:196] Error preparing data for projected volume kube-api-access-rj5dx for pod openshift-network-diagnostics/network-check-target-jgp54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873132    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx podName:dd75c1a3-b9c4-4300-928c-efe3b8c85ee4 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:23.873115888 +0000 UTC m=+66.289818641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj5dx" (UniqueName: "kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx") pod "network-check-target-jgp54" (UID: "dd75c1a3-b9c4-4300-928c-efe3b8c85ee4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:51.873163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873245    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:51.873435 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:51.873275    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:23.873265441 +0000 UTC m=+66.289968193 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:53.118006 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.117522    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:53.118006 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.117522    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:53.118006 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.117661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:53.118006 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:53.117660    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ps6t" podUID="2cb5a23e-f4e8-4542-b175-5fe7be200604"
Sep 16 18:43:53.118006 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:53.117735    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgp54" podUID="dd75c1a3-b9c4-4300-928c-efe3b8c85ee4"
Sep 16 18:43:53.118006 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:53.117816    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89j2l" podUID="4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab"
Sep 16 18:43:53.921225 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.921191    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-180.ec2.internal" event="NodeReady"
Sep 16 18:43:53.921419 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.921355    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:43:53.958075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.957993    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59b5bc4865-7zbhp"]
Sep 16 18:43:53.962841 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.962814    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:53.965186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.965163    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:43:53.965186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.965186    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wb8gd\""
Sep 16 18:43:53.965450 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.965341    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:43:53.965691 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.965661    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:43:53.971603 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.971583    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:43:53.975400 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.975380    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cfxqx"]
Sep 16 18:43:53.978561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.978539    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:53.979893 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.979869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b5bc4865-7zbhp"]
Sep 16 18:43:53.980627 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.980607    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zq67w"]
Sep 16 18:43:53.981738 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.981515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:43:53.981738 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.981550    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6mpm2\""
Sep 16 18:43:53.981738 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.981602    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:43:53.983743 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.983728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:53.985693 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.985677    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:43:53.986401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.986240    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nvn9q\""
Sep 16 18:43:53.987104 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.987087    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:43:53.987210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.987137    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:43:53.991340 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.991302    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cfxqx"]
Sep 16 18:43:53.994869 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:53.994851    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq67w"]
Sep 16 18:43:54.089669 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2740806-51d2-4432-a42d-384de809b042-tmp-dir\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.089851 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-image-registry-private-configuration\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.089851 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2740806-51d2-4432-a42d-384de809b042-config-volume\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.089851 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.089851 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-ca-trust-extracted\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.089851 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278sf\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-kube-api-access-278sf\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-trusted-ca\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-installation-pull-secrets\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcpz\" (UniqueName: \"kubernetes.io/projected/80e76374-0e57-4816-9207-e5837032f7c1-kube-api-access-mfcpz\") pod \"ingress-canary-zq67w\" (UID: \"80e76374-0e57-4816-9207-e5837032f7c1\") " pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.089997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-bound-sa-token\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.090036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80e76374-0e57-4816-9207-e5837032f7c1-cert\") pod \"ingress-canary-zq67w\" (UID: \"80e76374-0e57-4816-9207-e5837032f7c1\") " pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.090077 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.090059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5sh\" (UniqueName: \"kubernetes.io/projected/f2740806-51d2-4432-a42d-384de809b042-kube-api-access-5j5sh\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.090293 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.090100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-certificates\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.147092 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.147061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:54.191192 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2740806-51d2-4432-a42d-384de809b042-tmp-dir\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-image-registry-private-configuration\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2740806-51d2-4432-a42d-384de809b042-config-volume\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-ca-trust-extracted\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278sf\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-kube-api-access-278sf\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-trusted-ca\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-installation-pull-secrets\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcpz\" (UniqueName: \"kubernetes.io/projected/80e76374-0e57-4816-9207-e5837032f7c1-kube-api-access-mfcpz\") pod \"ingress-canary-zq67w\" (UID: \"80e76374-0e57-4816-9207-e5837032f7c1\") " pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-bound-sa-token\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80e76374-0e57-4816-9207-e5837032f7c1-cert\") pod \"ingress-canary-zq67w\" (UID: \"80e76374-0e57-4816-9207-e5837032f7c1\") " pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5sh\" (UniqueName: \"kubernetes.io/projected/f2740806-51d2-4432-a42d-384de809b042-kube-api-access-5j5sh\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-certificates\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.191763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2740806-51d2-4432-a42d-384de809b042-tmp-dir\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.192104 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.191969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-certificates\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.192557 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.192524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-trusted-ca\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.192695 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.192648    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:54.192695 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.192667    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:43:54.192783 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.192736    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.692717526 +0000 UTC m=+37.109420268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:43:54.193314 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.193290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2740806-51d2-4432-a42d-384de809b042-config-volume\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.193433 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.193424    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:54.193538 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.193523    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.693505629 +0000 UTC m=+37.110208379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:54.193654 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.193637    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-ca-trust-extracted\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.197135 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.197032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-image-registry-private-configuration\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.197232 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.197084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-installation-pull-secrets\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.197232 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.197112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80e76374-0e57-4816-9207-e5837032f7c1-cert\") pod \"ingress-canary-zq67w\" (UID: \"80e76374-0e57-4816-9207-e5837032f7c1\") " pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.203965 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.203937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5sh\" (UniqueName: \"kubernetes.io/projected/f2740806-51d2-4432-a42d-384de809b042-kube-api-access-5j5sh\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.204207 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.204177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-bound-sa-token\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.204297 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.204214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcpz\" (UniqueName: \"kubernetes.io/projected/80e76374-0e57-4816-9207-e5837032f7c1-kube-api-access-mfcpz\") pod \"ingress-canary-zq67w\" (UID: \"80e76374-0e57-4816-9207-e5837032f7c1\") " pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.206747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.206721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-278sf\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-kube-api-access-278sf\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.249117 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.249034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:54.298065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.298038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq67w"
Sep 16 18:43:54.584297 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.584271    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq67w"]
Sep 16 18:43:54.587091 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:43:54.587056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e76374_0e57_4816_9207_e5837032f7c1.slice/crio-84c070963b193b83cf82120898e4445309cdbe0dea83e18cd5f3fa240404ff67 WatchSource:0}: Error finding container 84c070963b193b83cf82120898e4445309cdbe0dea83e18cd5f3fa240404ff67: Status 404 returned error can't find the container with id 84c070963b193b83cf82120898e4445309cdbe0dea83e18cd5f3fa240404ff67
Sep 16 18:43:54.695976 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.695945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:54.696118 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:54.696007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:54.696118 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.696108    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:54.696118 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.696119    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:43:54.696258 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.696116    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:54.696258 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.696168    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:55.696155484 +0000 UTC m=+38.112858221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:43:54.696258 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:54.696197    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:55.696178027 +0000 UTC m=+38.112880778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:55.117608 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.117574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:43:55.117608 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.117614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:43:55.117854 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.117574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:43:55.120805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.120650    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:43:55.120805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.120667    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:43:55.120805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.120717    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7bds\""
Sep 16 18:43:55.120805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.120734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:43:55.120805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.120667    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:43:55.121126 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.120900    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bb8r2\""
Sep 16 18:43:55.356114 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.356075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq67w" event={"ID":"80e76374-0e57-4816-9207-e5837032f7c1","Type":"ContainerStarted","Data":"84c070963b193b83cf82120898e4445309cdbe0dea83e18cd5f3fa240404ff67"}
Sep 16 18:43:55.359473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.359448    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c9ced61-ba03-421f-bda1-20977b70864c" containerID="4b9df0bdfecb6040b79b174a7297b6e489a276ddb06ff5269efbc15b3e09ede0" exitCode=0
Sep 16 18:43:55.359598 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.359489    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerDied","Data":"4b9df0bdfecb6040b79b174a7297b6e489a276ddb06ff5269efbc15b3e09ede0"}
Sep 16 18:43:55.703602 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.703521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:55.703773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:55.703611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:55.703773 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:55.703699    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:55.703773 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:55.703746    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:55.703773 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:55.703761    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:43:55.703773 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:55.703775    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:57.703758775 +0000 UTC m=+40.120461535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:55.703990 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:55.703808    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:57.703793644 +0000 UTC m=+40.120496413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:43:56.326926 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:56.326885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:56.366181 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:56.366145    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c9ced61-ba03-421f-bda1-20977b70864c" containerID="12f4be0ed1cf34babf46d7378496ea642af9bc512eafca5cfb9191b2fcf14c1f" exitCode=0
Sep 16 18:43:56.366631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:56.366203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerDied","Data":"12f4be0ed1cf34babf46d7378496ea642af9bc512eafca5cfb9191b2fcf14c1f"}
Sep 16 18:43:56.414454 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:56.414417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:57.369824 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.369601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq67w" event={"ID":"80e76374-0e57-4816-9207-e5837032f7c1","Type":"ContainerStarted","Data":"33943b34ee192c004a1e8e2d3f150bb9527e661176712d3c4af8942c0c4ca3e5"}
Sep 16 18:43:57.373063 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.373032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" event={"ID":"3c9ced61-ba03-421f-bda1-20977b70864c","Type":"ContainerStarted","Data":"bc9f845561dca4bd6ad1e6d0d565a04fe2ff2857bb7088dd901e5d6c97418d19"}
Sep 16 18:43:57.388333 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.388274    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zq67w" podStartSLOduration=2.028351996 podStartE2EDuration="4.388259811s" totalImagesPullingTime="2.359907815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:57.387488727 +0000 UTC m=+39.804191486" watchObservedRunningTime="2026-09-16 18:43:57.388259811 +0000 UTC m=+39.804962593"
Sep 16 18:43:57.411512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.411467    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mm2fq" podStartSLOduration=9.092684803 podStartE2EDuration="38.41145192s" totalImagesPullingTime="29.318767117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:57.409227419 +0000 UTC m=+39.825930197" watchObservedRunningTime="2026-09-16 18:43:57.41145192 +0000 UTC m=+39.828154680"
Sep 16 18:43:57.537955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.537921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:57.641821 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.641750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:57.722004 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.721979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:43:57.722087 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:57.722042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:43:57.722139 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:57.722132    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:43:57.722174 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:57.722136    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:43:57.722174 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:57.722151    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:43:57.722242 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:57.722189    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:01.722174408 +0000 UTC m=+44.138877150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:43:57.722242 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:43:57.722203    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:01.722197492 +0000 UTC m=+44.138900229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:43:58.352483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:58.352449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:58.454009 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:58.453976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:59.708304 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:59.708271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:59.805914 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:43:59.805883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:01.623003 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:01.622966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:01.724460 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:01.724423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:01.753856 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:01.753823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:44:01.753994 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:01.753909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:01.753994 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:01.753980    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:01.754075 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:01.754047    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:01.754075 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:01.754062    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:44:01.754141 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:01.754055    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:09.754038643 +0000 UTC m=+52.170741382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:01.754141 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:01.754112    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:09.75409871 +0000 UTC m=+52.170801447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:44:04.799889 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:04.799856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:04.897756 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:04.897734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:06.654547 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:06.654516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:06.758881 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:06.758851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:07.663241 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:07.663205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:07.725451 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:07.725423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:09.816063 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:09.816024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:09.816470 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:09.816080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:44:09.816470 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:09.816174    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:09.816470 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:09.816180    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:09.816470 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:09.816198    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:44:09.816470 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:09.816230    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.816214791 +0000 UTC m=+68.232917529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:09.816470 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:09.816243    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.816237785 +0000 UTC m=+68.232940521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:44:10.614662 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:10.614632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:11.344247 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.344218    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-gn5r4"]
Sep 16 18:44:11.346293 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.346272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.351020 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.351001    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.351469 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.351452    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.351608 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.351594    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:44:11.351779 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.351767    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:44:11.351841 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.351769    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hn45g\""
Sep 16 18:44:11.359163 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.359143    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:44:11.364105 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.364085    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gn5r4"]
Sep 16 18:44:11.526573 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.526542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-snapshots\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.526573 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.526579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-serving-cert\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.526837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.526667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-tmp\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.526837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.526708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-service-ca-bundle\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.526837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.526756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.526837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.526777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9s9x\" (UniqueName: \"kubernetes.io/projected/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-kube-api-access-h9s9x\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.563724 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.563698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:11.627261 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627261 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9s9x\" (UniqueName: \"kubernetes.io/projected/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-kube-api-access-h9s9x\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627261 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-snapshots\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627480 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-serving-cert\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627529 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-tmp\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627571 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-service-ca-bundle\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627857 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-tmp\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.627978 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.627880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-snapshots\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.628071 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.628054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-service-ca-bundle\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.628127 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.628108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.631048 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.631022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-serving-cert\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.641851 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.641832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9s9x\" (UniqueName: \"kubernetes.io/projected/a180ce87-cf24-4c64-bbc8-0dd2ae85013f-kube-api-access-h9s9x\") pod \"insights-operator-5844b856db-gn5r4\" (UID: \"a180ce87-cf24-4c64-bbc8-0dd2ae85013f\") " pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.656168 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.656145    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gn5r4"
Sep 16 18:44:11.701025 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.700983    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:11.929349 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:11.929248    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gn5r4"]
Sep 16 18:44:11.934495 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:11.934470    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda180ce87_cf24_4c64_bbc8_0dd2ae85013f.slice/crio-91b351a7bf80d2feae2d13543bfcf80d8ff746f8bd45ca6e07032457eee3a4e2 WatchSource:0}: Error finding container 91b351a7bf80d2feae2d13543bfcf80d8ff746f8bd45ca6e07032457eee3a4e2: Status 404 returned error can't find the container with id 91b351a7bf80d2feae2d13543bfcf80d8ff746f8bd45ca6e07032457eee3a4e2
Sep 16 18:44:12.408522 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:12.408476    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gn5r4" event={"ID":"a180ce87-cf24-4c64-bbc8-0dd2ae85013f","Type":"ContainerStarted","Data":"91b351a7bf80d2feae2d13543bfcf80d8ff746f8bd45ca6e07032457eee3a4e2"}
Sep 16 18:44:14.415284 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:14.415248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gn5r4" event={"ID":"a180ce87-cf24-4c64-bbc8-0dd2ae85013f","Type":"ContainerStarted","Data":"b15e05ea65dc1076d78687908d58811f3d36ae4ee2e173833595e7a2d4ac5a0f"}
Sep 16 18:44:14.433279 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:14.433223    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-gn5r4" podStartSLOduration=1.735120124 podStartE2EDuration="3.433202588s" totalImagesPullingTime="1.698082464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:14.432882583 +0000 UTC m=+56.849585342" watchObservedRunningTime="2026-09-16 18:44:14.433202588 +0000 UTC m=+56.849905348"
Sep 16 18:44:14.999630 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:14.999599    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m4jcd"]
Sep 16 18:44:15.001541 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.001526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.003744 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.003721    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:44:15.003856 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.003748    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:44:15.004622 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.004559    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlsxc\""
Sep 16 18:44:15.012723 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.012694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4jcd"]
Sep 16 18:44:15.017670 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.017628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:15.051881 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.051847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.052012 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.051883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8lt\" (UniqueName: \"kubernetes.io/projected/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-api-access-lt8lt\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.052012 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.051956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-crio-socket\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.052012 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.052006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-data-volume\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.052146 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.052031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.089196 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.089166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:15.152747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.152717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-data-volume\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.152872 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.152756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.152872 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.152836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.152872 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.152864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8lt\" (UniqueName: \"kubernetes.io/projected/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-api-access-lt8lt\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.153013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.152905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-crio-socket\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.153013 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.152921    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.153013 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.152982    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:15.153013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.152990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-crio-socket\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.153013 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.153007    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.652983757 +0000 UTC m=+58.069686495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.153208 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.153027    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:15.653015428 +0000 UTC m=+58.069718198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:15.153208 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.153124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-data-volume\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.161352 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.161311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8lt\" (UniqueName: \"kubernetes.io/projected/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-api-access-lt8lt\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.656489 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.656450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.656885 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:15.656517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:15.656885 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.656585    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.656885 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.656622    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:15.656885 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.656637    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:16.656624012 +0000 UTC m=+59.073326749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:15.656885 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:15.656688    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:16.656669415 +0000 UTC m=+59.073372175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:16.179211 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.179176    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"]
Sep 16 18:44:16.182778 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.182758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.185521 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.185501    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:44:16.185632 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.185536    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gm476\""
Sep 16 18:44:16.185850 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.185832    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:16.186518 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.185967    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:16.193621 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.193601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"]
Sep 16 18:44:16.261927 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.261898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpf7\" (UniqueName: \"kubernetes.io/projected/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-kube-api-access-jlpf7\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.262109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.262040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.362903 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.362862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.363075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.362935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpf7\" (UniqueName: \"kubernetes.io/projected/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-kube-api-access-jlpf7\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.363075 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.363008    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:16.363075 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.363073    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls podName:ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:16.863057456 +0000 UTC m=+59.279760193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9qxj7" (UID: "ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d") : secret "samples-operator-tls" not found
Sep 16 18:44:16.371723 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.371694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpf7\" (UniqueName: \"kubernetes.io/projected/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-kube-api-access-jlpf7\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.548548 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.548519    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmdz7_3f93bc5c-ba37-4cbc-8c8e-9062254a738c/dns-node-resolver/0.log"
Sep 16 18:44:16.664702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.664668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:16.665122 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.664744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:16.665122 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.664812    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:16.665122 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.664836    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:16.665122 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.664892    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:18.664879285 +0000 UTC m=+61.081582022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:16.665122 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.664905    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:18.664898334 +0000 UTC m=+61.081601071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:16.866914 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:16.866831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:16.867071 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.866996    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:16.867071 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:16.867064    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls podName:ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:17.867049304 +0000 UTC m=+60.283752041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9qxj7" (UID: "ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d") : secret "samples-operator-tls" not found
Sep 16 18:44:17.148458 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:17.148381    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4m88x_572231be-7380-44a9-8379-caf0f42c820d/node-ca/0.log"
Sep 16 18:44:17.874616 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:17.874579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:17.875092 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:17.874720    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:17.875092 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:17.874804    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls podName:ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:19.874785666 +0000 UTC m=+62.291488420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9qxj7" (UID: "ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d") : secret "samples-operator-tls" not found
Sep 16 18:44:18.147969 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.147903    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq67w_80e76374-0e57-4816-9207-e5837032f7c1/serve-healthcheck-canary/0.log"
Sep 16 18:44:18.534957 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.534867    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.612198 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.612168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.682021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.681988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:18.682193 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.682100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:18.682193 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:18.682107    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:18.682193 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:18.682176    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.682160875 +0000 UTC m=+65.098863611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:18.682373 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:18.682239    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:18.682373 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:18.682289    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:22.682277585 +0000 UTC m=+65.098980323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:18.903044 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.903009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.960463 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:18.960433    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.349737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:19.349711    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lzd4j"
Sep 16 18:44:19.509217 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:19.509183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.609528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:19.609447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.894132 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:19.894042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:19.894272 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:19.894162    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:19.894272 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:19.894220    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls podName:ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:23.894206234 +0000 UTC m=+66.310908971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9qxj7" (UID: "ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d") : secret "samples-operator-tls" not found
Sep 16 18:44:20.751514 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:20.751478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.828644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:20.828617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.181272 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.181239    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"]
Sep 16 18:44:21.184670 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.184654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.186949 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.186928    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:44:21.187062 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.186977    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:44:21.187146 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.187129    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4kb5q\""
Sep 16 18:44:21.187208 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.187131    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:21.187708 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.187692    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:21.192868 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.192849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"]
Sep 16 18:44:21.205908 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.205885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec65898b-303d-405e-9830-5e9d0a05c616-config\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.206011 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.205923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8ls\" (UniqueName: \"kubernetes.io/projected/ec65898b-303d-405e-9830-5e9d0a05c616-kube-api-access-lt8ls\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.206011 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.205952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec65898b-303d-405e-9830-5e9d0a05c616-serving-cert\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.306517 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.306477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec65898b-303d-405e-9830-5e9d0a05c616-config\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.306696 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.306536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8ls\" (UniqueName: \"kubernetes.io/projected/ec65898b-303d-405e-9830-5e9d0a05c616-kube-api-access-lt8ls\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.306696 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.306572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec65898b-303d-405e-9830-5e9d0a05c616-serving-cert\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.307064 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.307044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec65898b-303d-405e-9830-5e9d0a05c616-config\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.308893 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.308873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec65898b-303d-405e-9830-5e9d0a05c616-serving-cert\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.315211 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.315188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8ls\" (UniqueName: \"kubernetes.io/projected/ec65898b-303d-405e-9830-5e9d0a05c616-kube-api-access-lt8ls\") pod \"service-ca-operator-cf7c776b7-jtxxg\" (UID: \"ec65898b-303d-405e-9830-5e9d0a05c616\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.494228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.494133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"
Sep 16 18:44:21.761828 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:21.761752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg"]
Sep 16 18:44:21.765435 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:21.765408    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec65898b_303d_405e_9830_5e9d0a05c616.slice/crio-ba840a917e1b2b1af38a0fa014991a2e2ae22e9e897043914a8aebea6947bb9d WatchSource:0}: Error finding container ba840a917e1b2b1af38a0fa014991a2e2ae22e9e897043914a8aebea6947bb9d: Status 404 returned error can't find the container with id ba840a917e1b2b1af38a0fa014991a2e2ae22e9e897043914a8aebea6947bb9d
Sep 16 18:44:22.438604 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:22.438518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg" event={"ID":"ec65898b-303d-405e-9830-5e9d0a05c616","Type":"ContainerStarted","Data":"ba840a917e1b2b1af38a0fa014991a2e2ae22e9e897043914a8aebea6947bb9d"}
Sep 16 18:44:22.718130 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:22.718047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:22.718267 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:22.718135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:22.718267 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:22.718211    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:22.718267 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:22.718229    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:22.718407 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:22.718288    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:30.718268025 +0000 UTC m=+73.134970764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:22.718407 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:22.718308    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:30.718299542 +0000 UTC m=+73.135002280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:23.929760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.929718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:44:23.929760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.929760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:44:23.930347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.929778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:23.930347 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:23.929890    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:23.930347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.929896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:44:23.930347 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:23.929941    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls podName:ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:31.929927516 +0000 UTC m=+74.346630253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9qxj7" (UID: "ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d") : secret "samples-operator-tls" not found
Sep 16 18:44:23.933090 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.933070    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:23.933258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.933242    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:23.933570 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.933558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:23.940131 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:23.940112    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:44:23.940210 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:23.940153    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs podName:2cb5a23e-f4e8-4542-b175-5fe7be200604 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:27.940142224 +0000 UTC m=+130.356844961 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs") pod "network-metrics-daemon-5ps6t" (UID: "2cb5a23e-f4e8-4542-b175-5fe7be200604") : secret "metrics-daemon-secret" not found
Sep 16 18:44:23.942210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.942190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab-original-pull-secret\") pod \"global-pull-secret-syncer-89j2l\" (UID: \"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab\") " pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:44:23.942292 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.942232    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:23.952489 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:23.952469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5dx\" (UniqueName: \"kubernetes.io/projected/dd75c1a3-b9c4-4300-928c-efe3b8c85ee4-kube-api-access-rj5dx\") pod \"network-check-target-jgp54\" (UID: \"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4\") " pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:44:24.229642 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.229561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89j2l"
Sep 16 18:44:24.239285 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.239258    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bb8r2\""
Sep 16 18:44:24.247546 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.247528    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:44:24.340028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.340005    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:24.407803 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.407776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:24.445480 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.445454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg" event={"ID":"ec65898b-303d-405e-9830-5e9d0a05c616","Type":"ContainerStarted","Data":"3feeccf68022c805c17d7a02f12d22fcc6c55ff347099a33658266390329f380"}
Sep 16 18:44:24.459816 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.459766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg" podStartSLOduration=1.154266555 podStartE2EDuration="3.459751072s" totalImagesPullingTime="2.305484517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:24.459229842 +0000 UTC m=+66.875932600" watchObservedRunningTime="2026-09-16 18:44:24.459751072 +0000 UTC m=+66.876453831"
Sep 16 18:44:24.517417 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.517385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89j2l"]
Sep 16 18:44:24.520623 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:24.520597    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b12fc6d_a87c_42cd_be81_b8ff0f0d4fab.slice/crio-1f2fb099a66bd117d79a8124edb4b476e58cb357f67933813808980d617257d6 WatchSource:0}: Error finding container 1f2fb099a66bd117d79a8124edb4b476e58cb357f67933813808980d617257d6: Status 404 returned error can't find the container with id 1f2fb099a66bd117d79a8124edb4b476e58cb357f67933813808980d617257d6
Sep 16 18:44:24.546132 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:24.546094    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jgp54"]
Sep 16 18:44:24.549621 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:24.549599    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd75c1a3_b9c4_4300_928c_efe3b8c85ee4.slice/crio-aee84700d2511e786e024dda54ba3ea19943f403188a7391fe8db3ea8599ec85 WatchSource:0}: Error finding container aee84700d2511e786e024dda54ba3ea19943f403188a7391fe8db3ea8599ec85: Status 404 returned error can't find the container with id aee84700d2511e786e024dda54ba3ea19943f403188a7391fe8db3ea8599ec85
Sep 16 18:44:25.449978 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:25.449937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89j2l" event={"ID":"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab","Type":"ContainerStarted","Data":"1f2fb099a66bd117d79a8124edb4b476e58cb357f67933813808980d617257d6"}
Sep 16 18:44:25.451829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:25.451782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jgp54" event={"ID":"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4","Type":"ContainerStarted","Data":"aee84700d2511e786e024dda54ba3ea19943f403188a7391fe8db3ea8599ec85"}
Sep 16 18:44:25.511955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:25.511920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:25.578240 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:25.578207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:25.850022 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:25.849982    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:25.850214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:25.850053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:44:25.850278 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:25.850234    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:25.850366 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:25.850296    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls podName:f2740806-51d2-4432-a42d-384de809b042 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.850277903 +0000 UTC m=+100.266980659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls") pod "dns-default-cfxqx" (UID: "f2740806-51d2-4432-a42d-384de809b042") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:25.850444 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:25.850405    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:25.850444 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:25.850419    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b5bc4865-7zbhp: secret "image-registry-tls" not found
Sep 16 18:44:25.850536 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:25.850458    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls podName:07599eaa-c564-4ecd-81d3-6bd4a70ebc75 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.850446407 +0000 UTC m=+100.267149145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls") pod "image-registry-59b5bc4865-7zbhp" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75") : secret "image-registry-tls" not found
Sep 16 18:44:26.112788 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:26.112710    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.194616 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:26.194578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.554149 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:26.554017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.675256 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:26.675222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.858632 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:26.858501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.958008 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:26.957919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.150201 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:28.150142    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.376912 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:28.376876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.463501 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:28.463418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.574643 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:28.574600    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.375247 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:29.375215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.463922 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:29.463892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89j2l" event={"ID":"4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab","Type":"ContainerStarted","Data":"7a8dbe19ed9a253446b6b8c48a3c1d903999686bac96bcb6e9cb5998a620cb4f"}
Sep 16 18:44:29.465508 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:29.465480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jgp54" event={"ID":"dd75c1a3-b9c4-4300-928c-efe3b8c85ee4","Type":"ContainerStarted","Data":"2c2d1a452cae26a2ef397382ce9efac6d30e8d557afe30ccd18c99963131e65b"}
Sep 16 18:44:29.465627 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:29.465614    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:44:29.481091 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:29.481048    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-89j2l" podStartSLOduration=66.020246262 podStartE2EDuration="1m10.481035851s" totalImagesPullingTime="4.460789589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:29.480223857 +0000 UTC m=+71.896926629" watchObservedRunningTime="2026-09-16 18:44:29.481035851 +0000 UTC m=+71.897738658"
Sep 16 18:44:29.497232 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:29.497187    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jgp54" podStartSLOduration=67.067968968 podStartE2EDuration="1m11.497170909s" totalImagesPullingTime="4.429201941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:29.496911213 +0000 UTC m=+71.913613974" watchObservedRunningTime="2026-09-16 18:44:29.497170909 +0000 UTC m=+71.913873668"
Sep 16 18:44:30.792669 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:30.792635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:30.793030 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:30.792703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:30.793030 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:30.792789    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:30.793030 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:30.792860    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls podName:a181ba71-9d0f-4556-8a94-f7ee21ed9fc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:46.79284389 +0000 UTC m=+89.209546633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4jcd" (UID: "a181ba71-9d0f-4556-8a94-f7ee21ed9fc3") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:30.793213 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:30.793193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:32.003523 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:32.003474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:32.003909 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:32.003630    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:44:32.003909 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:44:32.003697    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls podName:ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d nodeName:}" failed. No retries permitted until 2026-09-16 18:44:48.003682113 +0000 UTC m=+90.420384850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-9qxj7" (UID: "ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d") : secret "samples-operator-tls" not found
Sep 16 18:44:32.593129 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:32.593099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:32.697999 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:32.697928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:32.817482 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:32.817447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.012162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:33.012075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.127688 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:33.127653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.234334 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:33.234268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.336029 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:33.335971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.491756 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:33.491726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.689724 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:33.689648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:34.487542 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:34.487500    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:34.689193 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:34.689158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.092252 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:35.092219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.885447 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:35.885392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.195727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:36.195617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.296661 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:36.296627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.351099 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:36.351069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.486381 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:36.486279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.691575 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:36.691540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.085831 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:37.085801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.151545 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:37.151515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.886137 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:37.886099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.297225 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:38.297135    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.488553 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:38.488523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.690956 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:38.690921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:38.789523 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:38.789491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:39.168763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:39.168729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:39.492454 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:39.492362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.091549 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:40.091508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.359558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:40.359467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.692874 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:40.692780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.892357 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:40.892303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.123403 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:41.123361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.312437 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:41.312407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.392720 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:41.392643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.489392 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:41.489364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.084555 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:42.084521    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.490869 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:42.490791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.895041 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:42.895011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:43.088821 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:43.088788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:43.295272 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:43.295187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.811891 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:45.811858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.961458 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:45.961429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:46.820572 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:46.820539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:46.823012 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:46.822994    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a181ba71-9d0f-4556-8a94-f7ee21ed9fc3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4jcd\" (UID: \"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3\") " pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:46.991660 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:46.991616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.115127 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:47.115045    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlsxc\""
Sep 16 18:44:47.122636 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:47.122609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4jcd"
Sep 16 18:44:47.391430 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:47.391353    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4jcd"]
Sep 16 18:44:47.394214 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:47.394185    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda181ba71_9d0f_4556_8a94_f7ee21ed9fc3.slice/crio-f1af918dbdad9259269d98b987c7eb11cdca25a77d37d0b832011926e1021915 WatchSource:0}: Error finding container f1af918dbdad9259269d98b987c7eb11cdca25a77d37d0b832011926e1021915: Status 404 returned error can't find the container with id f1af918dbdad9259269d98b987c7eb11cdca25a77d37d0b832011926e1021915
Sep 16 18:44:47.516151 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:47.516109    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4jcd" event={"ID":"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3","Type":"ContainerStarted","Data":"e5fc641d16c9b16b81e0982f3d0eefaf0fad1e09e0529247e4ffe8afd568b396"}
Sep 16 18:44:47.516344 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:47.516167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4jcd" event={"ID":"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3","Type":"ContainerStarted","Data":"f1af918dbdad9259269d98b987c7eb11cdca25a77d37d0b832011926e1021915"}
Sep 16 18:44:48.032443 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:48.032412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:48.035147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:48.035120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-9qxj7\" (UID: \"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:48.295730 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:48.295646    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gm476\""
Sep 16 18:44:48.303464 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:48.303438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"
Sep 16 18:44:48.522364 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:48.522308    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4jcd" event={"ID":"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3","Type":"ContainerStarted","Data":"908a6e2a3bbbe7dfd39b462ab7dac8bb88bdf3548507b0b90d437dbab0d27fb5"}
Sep 16 18:44:48.571130 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:48.571106    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7"]
Sep 16 18:44:49.528973 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:49.528892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7" event={"ID":"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d","Type":"ContainerStarted","Data":"e2e518394cdaf68a75a0a605c98aad096776e26d9d016127c10e584894ad0b8a"}
Sep 16 18:44:49.531461 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:49.531430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4jcd" event={"ID":"a181ba71-9d0f-4556-8a94-f7ee21ed9fc3","Type":"ContainerStarted","Data":"c05b89fd1c5df246cc41a897797ed572b722b035db52ab626a22829481a6e881"}
Sep 16 18:44:49.549419 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:49.549365    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m4jcd" podStartSLOduration=33.772527864 podStartE2EDuration="35.549345864s" totalImagesPullingTime="1.776818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:49.54852249 +0000 UTC m=+91.965225250" watchObservedRunningTime="2026-09-16 18:44:49.549345864 +0000 UTC m=+91.966048621"
Sep 16 18:44:50.213131 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.213059    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"]
Sep 16 18:44:50.217141 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.217121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.219514 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.219483    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g6bwt\""
Sep 16 18:44:50.220426 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.220395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:44:50.220525 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.220438    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:44:50.220725 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.220707    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:44:50.223677 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.223657    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:44:50.233797 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.233778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"]
Sep 16 18:44:50.350305 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.350272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4qk\" (UniqueName: \"kubernetes.io/projected/47930a0a-0752-428e-8ea3-beff4eead7fc-kube-api-access-9n4qk\") pod \"managed-serviceaccount-addon-agent-869574f4fb-v6ssb\" (UID: \"47930a0a-0752-428e-8ea3-beff4eead7fc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.350505 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.350355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47930a0a-0752-428e-8ea3-beff4eead7fc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869574f4fb-v6ssb\" (UID: \"47930a0a-0752-428e-8ea3-beff4eead7fc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.377106 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.377066    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"]
Sep 16 18:44:50.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.379349    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.383702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.383495    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kqq2z\""
Sep 16 18:44:50.383702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.383666    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:44:50.383947 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.383867    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:44:50.398786 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.398752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"]
Sep 16 18:44:50.401223 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.401174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:50.451645 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.451617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4qk\" (UniqueName: \"kubernetes.io/projected/47930a0a-0752-428e-8ea3-beff4eead7fc-kube-api-access-9n4qk\") pod \"managed-serviceaccount-addon-agent-869574f4fb-v6ssb\" (UID: \"47930a0a-0752-428e-8ea3-beff4eead7fc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.451890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.451664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47930a0a-0752-428e-8ea3-beff4eead7fc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869574f4fb-v6ssb\" (UID: \"47930a0a-0752-428e-8ea3-beff4eead7fc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.454234 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.454213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47930a0a-0752-428e-8ea3-beff4eead7fc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869574f4fb-v6ssb\" (UID: \"47930a0a-0752-428e-8ea3-beff4eead7fc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.467507 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.467455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4qk\" (UniqueName: \"kubernetes.io/projected/47930a0a-0752-428e-8ea3-beff4eead7fc-kube-api-access-9n4qk\") pod \"managed-serviceaccount-addon-agent-869574f4fb-v6ssb\" (UID: \"47930a0a-0752-428e-8ea3-beff4eead7fc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.504642 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.504618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:50.549239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.549213    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"
Sep 16 18:44:50.554169 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.553971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-29v4m\" (UID: \"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.554169 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.554098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-29v4m\" (UID: \"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.609976 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.609945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:50.654716 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.654687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-29v4m\" (UID: \"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.654875 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.654847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-29v4m\" (UID: \"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.655388 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.655367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-29v4m\" (UID: \"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.657340 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.657294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-29v4m\" (UID: \"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.689472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.689444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"
Sep 16 18:44:50.826686 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.826654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb"]
Sep 16 18:44:50.829162 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:50.829139    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47930a0a_0752_428e_8ea3_beff4eead7fc.slice/crio-7be912b86e896c6de202e49ffcf49d965a7b086534bd5050541fa51f617e6b1f WatchSource:0}: Error finding container 7be912b86e896c6de202e49ffcf49d965a7b086534bd5050541fa51f617e6b1f: Status 404 returned error can't find the container with id 7be912b86e896c6de202e49ffcf49d965a7b086534bd5050541fa51f617e6b1f
Sep 16 18:44:50.954578 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:50.954550    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m"]
Sep 16 18:44:50.957487 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:50.957461    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f2a7cb_f471_4ef9_b4fb_40535d0d5a57.slice/crio-72ea358adfc3c1f479dff6889df7f4ec27800dc1812ae553ec5015148b032c96 WatchSource:0}: Error finding container 72ea358adfc3c1f479dff6889df7f4ec27800dc1812ae553ec5015148b032c96: Status 404 returned error can't find the container with id 72ea358adfc3c1f479dff6889df7f4ec27800dc1812ae553ec5015148b032c96
Sep 16 18:44:51.541220 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:51.541169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7" event={"ID":"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d","Type":"ContainerStarted","Data":"d1fa8dfd754c3a9364dd495f567e198869989d166310f5aa38a654b2f34ddec3"}
Sep 16 18:44:51.541401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:51.541279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7" event={"ID":"ad9d0b27-bddb-4425-8fdc-b5a0ab65f54d","Type":"ContainerStarted","Data":"774afd2229ee34aa442ee07870639d803cec266fb03802a5f2d9ab8667b8d38c"}
Sep 16 18:44:51.542710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:51.542685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m" event={"ID":"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57","Type":"ContainerStarted","Data":"72ea358adfc3c1f479dff6889df7f4ec27800dc1812ae553ec5015148b032c96"}
Sep 16 18:44:51.544003 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:51.543981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb" event={"ID":"47930a0a-0752-428e-8ea3-beff4eead7fc","Type":"ContainerStarted","Data":"7be912b86e896c6de202e49ffcf49d965a7b086534bd5050541fa51f617e6b1f"}
Sep 16 18:44:51.556746 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:51.556688    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-9qxj7" podStartSLOduration=33.69017568 podStartE2EDuration="35.556674177s" totalImagesPullingTime="1.866498497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.556169067 +0000 UTC m=+93.972871826" watchObservedRunningTime="2026-09-16 18:44:51.556674177 +0000 UTC m=+93.973376928"
Sep 16 18:44:52.549369 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:52.549334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m" event={"ID":"e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57","Type":"ContainerStarted","Data":"8f9bade97ab36ba8fdf1eee2f1f9aecd121bebee84d48d18389b66e17b544727"}
Sep 16 18:44:52.870751 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:52.870673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:52.966194 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:52.966159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:53.042346 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:53.042298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:53.185945 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:53.185859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:53.553560 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:53.553528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb" event={"ID":"47930a0a-0752-428e-8ea3-beff4eead7fc","Type":"ContainerStarted","Data":"00d28419b01edfbb6a5bbc97a232ddb24e0fa6e2cb94024311ae9deccd4a7a8d"}
Sep 16 18:44:53.568534 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:53.568489    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869574f4fb-v6ssb" podStartSLOduration=1.018651893 podStartE2EDuration="3.568473836s" totalImagesPullingTime="2.549821943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:53.567195114 +0000 UTC m=+95.983897874" watchObservedRunningTime="2026-09-16 18:44:53.568473836 +0000 UTC m=+95.985176595"
Sep 16 18:44:53.568652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:53.568559    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-29v4m" podStartSLOduration=2.376657138 podStartE2EDuration="3.568552767s" totalImagesPullingTime="1.191895629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:52.571849143 +0000 UTC m=+94.988551900" watchObservedRunningTime="2026-09-16 18:44:53.568552767 +0000 UTC m=+95.985255525"
Sep 16 18:44:56.301850 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:56.301815    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.363408 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:56.363375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:57.907689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:57.907652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:57.908153 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:57.907702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:44:57.910217 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:57.910198    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2740806-51d2-4432-a42d-384de809b042-metrics-tls\") pod \"dns-default-cfxqx\" (UID: \"f2740806-51d2-4432-a42d-384de809b042\") " pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:44:57.910361 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:57.910342    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"image-registry-59b5bc4865-7zbhp\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") " pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:58.177403 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.177302    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wb8gd\""
Sep 16 18:44:58.185095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.185074    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:58.192618 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.192597    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6mpm2\""
Sep 16 18:44:58.201463 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.201446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:44:58.490193 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.489996    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b5bc4865-7zbhp"]
Sep 16 18:44:58.492754 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:58.492729    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07599eaa_c564_4ecd_81d3_6bd4a70ebc75.slice/crio-dba485695294203eda55db3e21e543c6e8e854dce989d9682c74f0890fbeaaa5 WatchSource:0}: Error finding container dba485695294203eda55db3e21e543c6e8e854dce989d9682c74f0890fbeaaa5: Status 404 returned error can't find the container with id dba485695294203eda55db3e21e543c6e8e854dce989d9682c74f0890fbeaaa5
Sep 16 18:44:58.520044 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.520016    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cfxqx"]
Sep 16 18:44:58.522918 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:44:58.522889    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2740806_51d2_4432_a42d_384de809b042.slice/crio-db146ded3a26c00b6175cca19d91d98590f30bface3e2ffea19afaf3591bd4f8 WatchSource:0}: Error finding container db146ded3a26c00b6175cca19d91d98590f30bface3e2ffea19afaf3591bd4f8: Status 404 returned error can't find the container with id db146ded3a26c00b6175cca19d91d98590f30bface3e2ffea19afaf3591bd4f8
Sep 16 18:44:58.569794 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.569762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp" event={"ID":"07599eaa-c564-4ecd-81d3-6bd4a70ebc75","Type":"ContainerStarted","Data":"dba485695294203eda55db3e21e543c6e8e854dce989d9682c74f0890fbeaaa5"}
Sep 16 18:44:58.570701 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.570682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cfxqx" event={"ID":"f2740806-51d2-4432-a42d-384de809b042","Type":"ContainerStarted","Data":"db146ded3a26c00b6175cca19d91d98590f30bface3e2ffea19afaf3591bd4f8"}
Sep 16 18:44:58.939222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:58.939188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:59.000096 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:59.000061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:59.056352 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:59.056299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:59.576660 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:59.576620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp" event={"ID":"07599eaa-c564-4ecd-81d3-6bd4a70ebc75","Type":"ContainerStarted","Data":"47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d"}
Sep 16 18:44:59.576855 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:59.576749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:44:59.601125 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:59.601070    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp" podStartSLOduration=100.601051639 podStartE2EDuration="1m40.601051639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:59.600175548 +0000 UTC m=+102.016878318" watchObservedRunningTime="2026-09-16 18:44:59.601051639 +0000 UTC m=+102.017754397"
Sep 16 18:44:59.850272 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:44:59.850177    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.471208 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:00.471180    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jgp54"
Sep 16 18:45:00.583295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:00.583038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cfxqx" event={"ID":"f2740806-51d2-4432-a42d-384de809b042","Type":"ContainerStarted","Data":"2aa34a7ea2d358dcb59aa56d01f548eb47f0fbc1ffbc6e6581d3174cec0bfc1f"}
Sep 16 18:45:01.150111 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.150061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.328122 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.328088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.395173 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.395144    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"]
Sep 16 18:45:01.420414 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.420343    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.421208 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.421064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"]
Sep 16 18:45:01.424973 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.424949    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:45:01.425120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.424983    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:45:01.425120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.425040    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zxtqv\""
Sep 16 18:45:01.425224 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.425198    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:45:01.425277 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.425265    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:45:01.425434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.425408    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:45:01.448766 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.448739    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"]
Sep 16 18:45:01.470580 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.470553    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rtv7r"]
Sep 16 18:45:01.470695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.470666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.473659 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.473641    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:45:01.474072 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.474054    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:45:01.474141 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.474094    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lkxhd\""
Sep 16 18:45:01.474141 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.474111    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:45:01.490296 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.490277    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"]
Sep 16 18:45:01.490428 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.490398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.493011 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.492983    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:45:01.493120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.493078    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:45:01.493178 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.493124    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k6sj2\""
Sep 16 18:45:01.493360 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.493346    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:45:01.498173 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.498147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.533981 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.533955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100ace8a-d8dc-45ae-906e-c72d0cdca137-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.534098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.534020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100ace8a-d8dc-45ae-906e-c72d0cdca137-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.534098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.534050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfv8t\" (UniqueName: \"kubernetes.io/projected/100ace8a-d8dc-45ae-906e-c72d0cdca137-kube-api-access-bfv8t\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.534098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.534073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100ace8a-d8dc-45ae-906e-c72d0cdca137-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.560217 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.560186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.588782 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.588750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cfxqx" event={"ID":"f2740806-51d2-4432-a42d-384de809b042","Type":"ContainerStarted","Data":"a9432f5e3c3b5152bcb08d864bb47a7214fb6fc1b1a650464c458413fe165067"}
Sep 16 18:45:01.588913 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.588868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:45:01.606351 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.606285    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cfxqx" podStartSLOduration=66.682271319 podStartE2EDuration="1m8.606266665s" totalImagesPullingTime="1.923995346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.605451681 +0000 UTC m=+104.022154441" watchObservedRunningTime="2026-09-16 18:45:01.606266665 +0000 UTC m=+104.022969423"
Sep 16 18:45:01.634687 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100ace8a-d8dc-45ae-906e-c72d0cdca137-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.634837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.634837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.634837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-textfile\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.634837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-tls\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.634997 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100ace8a-d8dc-45ae-906e-c72d0cdca137-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.634997 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634906    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53da9125-8892-4a1b-9bd8-c5974d12cf52-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.634997 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-wtmp\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.634997 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea21f56-d2d5-41af-89eb-4b370def1c75-metrics-client-ca\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.635147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.634999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-sys\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.635147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.635147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfv8t\" (UniqueName: \"kubernetes.io/projected/100ace8a-d8dc-45ae-906e-c72d0cdca137-kube-api-access-bfv8t\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.635147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-root\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.635147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-accelerators-collector-config\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.635452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100ace8a-d8dc-45ae-906e-c72d0cdca137-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.635452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8954\" (UniqueName: \"kubernetes.io/projected/0ea21f56-d2d5-41af-89eb-4b370def1c75-kube-api-access-s8954\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.635452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/53da9125-8892-4a1b-9bd8-c5974d12cf52-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.635452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvbj\" (UniqueName: \"kubernetes.io/projected/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-api-access-2dvbj\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.635452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.635840 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.635820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100ace8a-d8dc-45ae-906e-c72d0cdca137-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.637414 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.637395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100ace8a-d8dc-45ae-906e-c72d0cdca137-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.637474 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.637395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100ace8a-d8dc-45ae-906e-c72d0cdca137-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.643303 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.643283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfv8t\" (UniqueName: \"kubernetes.io/projected/100ace8a-d8dc-45ae-906e-c72d0cdca137-kube-api-access-bfv8t\") pod \"openshift-state-metrics-67dcf8c979-mqknv\" (UID: \"100ace8a-d8dc-45ae-906e-c72d0cdca137\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.730345 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.730247    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"
Sep 16 18:45:01.736455 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53da9125-8892-4a1b-9bd8-c5974d12cf52-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.736609 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-wtmp\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736609 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea21f56-d2d5-41af-89eb-4b370def1c75-metrics-client-ca\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736609 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-sys\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736609 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736609 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-root\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-accelerators-collector-config\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8954\" (UniqueName: \"kubernetes.io/projected/0ea21f56-d2d5-41af-89eb-4b370def1c75-kube-api-access-s8954\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.736863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/53da9125-8892-4a1b-9bd8-c5974d12cf52-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.736863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvbj\" (UniqueName: \"kubernetes.io/projected/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-api-access-2dvbj\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.736863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.736863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.737186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.737186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-textfile\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.737186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.736979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-tls\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.737530 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.737468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-root\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.738080 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.737979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-sys\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.738882 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.738242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-accelerators-collector-config\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.738882 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.738389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53da9125-8892-4a1b-9bd8-c5974d12cf52-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.738882 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.738534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/53da9125-8892-4a1b-9bd8-c5974d12cf52-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.738882 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.738710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ea21f56-d2d5-41af-89eb-4b370def1c75-metrics-client-ca\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.738882 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.738840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-wtmp\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.739183 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.738936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-textfile\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.739264 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.739241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.740301 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.740256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-tls\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.740772 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.740743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ea21f56-d2d5-41af-89eb-4b370def1c75-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.741192 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.741167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.741566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.741548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.746039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.746007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvbj\" (UniqueName: \"kubernetes.io/projected/53da9125-8892-4a1b-9bd8-c5974d12cf52-kube-api-access-2dvbj\") pod \"kube-state-metrics-59fbdd77d-pdwkp\" (UID: \"53da9125-8892-4a1b-9bd8-c5974d12cf52\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.746116 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.746015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8954\" (UniqueName: \"kubernetes.io/projected/0ea21f56-d2d5-41af-89eb-4b370def1c75-kube-api-access-s8954\") pod \"node-exporter-rtv7r\" (UID: \"0ea21f56-d2d5-41af-89eb-4b370def1c75\") " pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.757095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.757050    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.779838 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.779805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"
Sep 16 18:45:01.799833 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.799804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rtv7r"
Sep 16 18:45:01.844431 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:01.844408    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea21f56_d2d5_41af_89eb_4b370def1c75.slice/crio-7ab63d76f90c319d583c38ceb17c824d867d26b38f6be5993da0e2d4820bd8fd WatchSource:0}: Error finding container 7ab63d76f90c319d583c38ceb17c824d867d26b38f6be5993da0e2d4820bd8fd: Status 404 returned error can't find the container with id 7ab63d76f90c319d583c38ceb17c824d867d26b38f6be5993da0e2d4820bd8fd
Sep 16 18:45:01.857093 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.857066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.986818 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:01.986730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:02.016952 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.016917    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv"]
Sep 16 18:45:02.020366 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:02.020339    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100ace8a_d8dc_45ae_906e_c72d0cdca137.slice/crio-b2a6c579610d11d46b3b4b9a1409df65fda44298e3955465195818012cd8eee2 WatchSource:0}: Error finding container b2a6c579610d11d46b3b4b9a1409df65fda44298e3955465195818012cd8eee2: Status 404 returned error can't find the container with id b2a6c579610d11d46b3b4b9a1409df65fda44298e3955465195818012cd8eee2
Sep 16 18:45:02.084969 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.084942    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp"]
Sep 16 18:45:02.090277 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:02.090246    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53da9125_8892_4a1b_9bd8_c5974d12cf52.slice/crio-9373625b563bea1c1e077f02b8fa181f2041d76f6b4e80aa414b7ca6dac0644e WatchSource:0}: Error finding container 9373625b563bea1c1e077f02b8fa181f2041d76f6b4e80aa414b7ca6dac0644e: Status 404 returned error can't find the container with id 9373625b563bea1c1e077f02b8fa181f2041d76f6b4e80aa414b7ca6dac0644e
Sep 16 18:45:02.395975 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.395939    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:02.480810 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.480761    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:45:02.483956 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.483931    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.484434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.484406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:02.486738 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.486713    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:45:02.486862 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.486789    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:45:02.486862 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.486815    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:45:02.487365 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:45:02.487476 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487412    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:45:02.487476 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:45:02.487701 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487648    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:45:02.488021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487707    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:45:02.488021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-skjk4\""
Sep 16 18:45:02.488021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.487831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:45:02.502900 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.502878    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:45:02.593934 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.593892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtv7r" event={"ID":"0ea21f56-d2d5-41af-89eb-4b370def1c75","Type":"ContainerStarted","Data":"7ab63d76f90c319d583c38ceb17c824d867d26b38f6be5993da0e2d4820bd8fd"}
Sep 16 18:45:02.595218 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.595183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp" event={"ID":"53da9125-8892-4a1b-9bd8-c5974d12cf52","Type":"ContainerStarted","Data":"9373625b563bea1c1e077f02b8fa181f2041d76f6b4e80aa414b7ca6dac0644e"}
Sep 16 18:45:02.597290 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.597260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv" event={"ID":"100ace8a-d8dc-45ae-906e-c72d0cdca137","Type":"ContainerStarted","Data":"50a06cd1f9ca2805c964d11a476b60710dee17dc6bf38717d731098acf1c0129"}
Sep 16 18:45:02.597430 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.597333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv" event={"ID":"100ace8a-d8dc-45ae-906e-c72d0cdca137","Type":"ContainerStarted","Data":"5994948c8802311296fa69f6213705181d8ac552edfc1e7ac470dcdf564086c5"}
Sep 16 18:45:02.597430 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.597366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv" event={"ID":"100ace8a-d8dc-45ae-906e-c72d0cdca137","Type":"ContainerStarted","Data":"b2a6c579610d11d46b3b4b9a1409df65fda44298e3955465195818012cd8eee2"}
Sep 16 18:45:02.644115 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644438 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644438 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644438 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644438 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8h9m\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-kube-api-access-j8h9m\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-web-config\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-out\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-volume\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.644824 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.644648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.744992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8h9m\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-kube-api-access-j8h9m\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-web-config\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-out\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-volume\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.745788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:45:02.745875    2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 18:45:02.746295 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:45:02.745935    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls podName:41f78a60-1e5e-484c-8dac-e7aa6b80de61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.245916473 +0000 UTC m=+105.662619286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61") : secret "alertmanager-main-tls" not found
Sep 16 18:45:02.747217 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:45:02.746479    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle podName:41f78a60-1e5e-484c-8dac-e7aa6b80de61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.246458961 +0000 UTC m=+105.663161712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61") : configmap references non-existent config key: ca-bundle.crt
Sep 16 18:45:02.747217 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.746661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.749219 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.749171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.755274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.754704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.755274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.755173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.755964 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.755918    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-out\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.758032 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.757993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-web-config\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.758517 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.758397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-volume\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.760475 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.760443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8h9m\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-kube-api-access-j8h9m\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.761991 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.761970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-tls-assets\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:02.764172 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:02.764133    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:03.250165 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.250117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:03.250392 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.250178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:03.250392 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:45:03.250306    2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 18:45:03.250509 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:45:03.250398    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls podName:41f78a60-1e5e-484c-8dac-e7aa6b80de61 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:04.250377872 +0000 UTC m=+106.667080613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61") : secret "alertmanager-main-tls" not found
Sep 16 18:45:03.251055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.251031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:03.444745 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.444709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:03.603249 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.603187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp" event={"ID":"53da9125-8892-4a1b-9bd8-c5974d12cf52","Type":"ContainerStarted","Data":"c1562c471b75b3304d0f809e79c988b208eecfa46796bdd854117160c84cee80"}
Sep 16 18:45:03.606927 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.606400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv" event={"ID":"100ace8a-d8dc-45ae-906e-c72d0cdca137","Type":"ContainerStarted","Data":"4c627aea3195c8bf5c77241c7d6af8d6d361a520176b64660dd6ce56931ae45e"}
Sep 16 18:45:03.608577 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.608535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtv7r" event={"ID":"0ea21f56-d2d5-41af-89eb-4b370def1c75","Type":"ContainerStarted","Data":"97bca27939f5c21452ce1c871d80085b80d95ef9c50cb6ca2e42bccf7fbf5be5"}
Sep 16 18:45:03.625414 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:03.625362    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mqknv" podStartSLOduration=1.275169068 podStartE2EDuration="2.625341674s" totalImagesPullingTime="1.350172606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:03.623856278 +0000 UTC m=+106.040559038" watchObservedRunningTime="2026-09-16 18:45:03.625341674 +0000 UTC m=+106.042044430"
Sep 16 18:45:04.259473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.259426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:04.261983 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.261964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:04.294883 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.294855    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:45:04.371951 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.371922    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:04.443873 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.443844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:04.576599 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.576560    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:45:04.578909 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:04.578890    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f78a60_1e5e_484c_8dac_e7aa6b80de61.slice/crio-8d8e5fa743a327f61ec90d4e142a964117bbd8b2e56cf38c83b89cdcb7bbf21d WatchSource:0}: Error finding container 8d8e5fa743a327f61ec90d4e142a964117bbd8b2e56cf38c83b89cdcb7bbf21d: Status 404 returned error can't find the container with id 8d8e5fa743a327f61ec90d4e142a964117bbd8b2e56cf38c83b89cdcb7bbf21d
Sep 16 18:45:04.613856 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.613828    2562 generic.go:356] "Generic (PLEG): container finished" podID="0ea21f56-d2d5-41af-89eb-4b370def1c75" containerID="97bca27939f5c21452ce1c871d80085b80d95ef9c50cb6ca2e42bccf7fbf5be5" exitCode=0
Sep 16 18:45:04.614237 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.613917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtv7r" event={"ID":"0ea21f56-d2d5-41af-89eb-4b370def1c75","Type":"ContainerDied","Data":"97bca27939f5c21452ce1c871d80085b80d95ef9c50cb6ca2e42bccf7fbf5be5"}
Sep 16 18:45:04.616249 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.616216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp" event={"ID":"53da9125-8892-4a1b-9bd8-c5974d12cf52","Type":"ContainerStarted","Data":"99568a3baf394decf34018c9c19cdcbeec73b8ea757eae12d690817b0cfe7ed7"}
Sep 16 18:45:04.616366 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.616313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp" event={"ID":"53da9125-8892-4a1b-9bd8-c5974d12cf52","Type":"ContainerStarted","Data":"1ffe55d411cb851ea63c592d86685a6dc9f0142ec70606d19590a6b95cf3e03f"}
Sep 16 18:45:04.617539 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.617507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"8d8e5fa743a327f61ec90d4e142a964117bbd8b2e56cf38c83b89cdcb7bbf21d"}
Sep 16 18:45:04.714506 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.714478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:04.806820 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:04.806784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:05.623511 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.623476    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtv7r" event={"ID":"0ea21f56-d2d5-41af-89eb-4b370def1c75","Type":"ContainerStarted","Data":"56e274aa3a316dc284dc17daef77f3d72271a46ce9e19c8e84cba79ea162e385"}
Sep 16 18:45:05.623924 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.623537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtv7r" event={"ID":"0ea21f56-d2d5-41af-89eb-4b370def1c75","Type":"ContainerStarted","Data":"9f331004208544287dbc2c22dce8ef02852a2af72d90755424fd698331a948b5"}
Sep 16 18:45:05.625060 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.625039    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb" exitCode=0
Sep 16 18:45:05.625153 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.625095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"}
Sep 16 18:45:05.642930 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.642892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-pdwkp" podStartSLOduration=3.232775381 podStartE2EDuration="4.642877829s" totalImagesPullingTime="1.410102448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:04.662811182 +0000 UTC m=+107.079513942" watchObservedRunningTime="2026-09-16 18:45:05.642877829 +0000 UTC m=+108.059580587"
Sep 16 18:45:05.643661 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.643633    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rtv7r" podStartSLOduration=2.884841434 podStartE2EDuration="4.643623757s" totalImagesPullingTime="1.653212835s" totalInitContainerRuntime="105.569488ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:05.641869007 +0000 UTC m=+108.058571763" watchObservedRunningTime="2026-09-16 18:45:05.643623757 +0000 UTC m=+108.060326516"
Sep 16 18:45:05.827999 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.827968    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bf57fd676-rx9x8"]
Sep 16 18:45:05.830139 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.830112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.832652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.832636    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:45:05.832806 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.832791    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:45:05.832907 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.832804    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:45:05.832907 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.832812    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-v9gpm\""
Sep 16 18:45:05.832907 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.832837    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:45:05.833075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.832923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-p7hnh03tne5i\""
Sep 16 18:45:05.839631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.839612    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bf57fd676-rx9x8"]
Sep 16 18:45:05.874015 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.873989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-client-ca-bundle\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.874133 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.874022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc57feab-c598-46a1-8d33-2ca94832f77f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.874133 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.874045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fc57feab-c598-46a1-8d33-2ca94832f77f-audit-log\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.874133 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.874064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rznw\" (UniqueName: \"kubernetes.io/projected/fc57feab-c598-46a1-8d33-2ca94832f77f-kube-api-access-2rznw\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.874236 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.874151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-secret-metrics-server-client-certs\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.874236 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.874174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-secret-metrics-server-tls\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.874236 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.874195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fc57feab-c598-46a1-8d33-2ca94832f77f-metrics-server-audit-profiles\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-client-ca-bundle\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc57feab-c598-46a1-8d33-2ca94832f77f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974875 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fc57feab-c598-46a1-8d33-2ca94832f77f-audit-log\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974875 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rznw\" (UniqueName: \"kubernetes.io/projected/fc57feab-c598-46a1-8d33-2ca94832f77f-kube-api-access-2rznw\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974875 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-secret-metrics-server-client-certs\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974875 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-secret-metrics-server-tls\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.974875 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.974758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fc57feab-c598-46a1-8d33-2ca94832f77f-metrics-server-audit-profiles\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.975270 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.975197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fc57feab-c598-46a1-8d33-2ca94832f77f-audit-log\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.975550 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.975471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc57feab-c598-46a1-8d33-2ca94832f77f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.975780 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.975760    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fc57feab-c598-46a1-8d33-2ca94832f77f-metrics-server-audit-profiles\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.977451 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.977422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-client-ca-bundle\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.977828 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.977804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-secret-metrics-server-tls\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.977897 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.977876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fc57feab-c598-46a1-8d33-2ca94832f77f-secret-metrics-server-client-certs\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:05.982578 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:05.982557    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rznw\" (UniqueName: \"kubernetes.io/projected/fc57feab-c598-46a1-8d33-2ca94832f77f-kube-api-access-2rznw\") pod \"metrics-server-7bf57fd676-rx9x8\" (UID: \"fc57feab-c598-46a1-8d33-2ca94832f77f\") " pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:06.139047 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:06.138794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:06.229147 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:06.229112    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:06.288531 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:06.288491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:06.419118 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:06.419020    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bf57fd676-rx9x8"]
Sep 16 18:45:06.422149 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:06.422114    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc57feab_c598_46a1_8d33_2ca94832f77f.slice/crio-662e675a57274e722a76a4b678106839c9fceb93ebbecbeb791958de926be33f WatchSource:0}: Error finding container 662e675a57274e722a76a4b678106839c9fceb93ebbecbeb791958de926be33f: Status 404 returned error can't find the container with id 662e675a57274e722a76a4b678106839c9fceb93ebbecbeb791958de926be33f
Sep 16 18:45:06.633217 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:06.633056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8" event={"ID":"fc57feab-c598-46a1-8d33-2ca94832f77f","Type":"ContainerStarted","Data":"662e675a57274e722a76a4b678106839c9fceb93ebbecbeb791958de926be33f"}
Sep 16 18:45:07.641245 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.641211    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:45:07.644404 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.644377    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"}
Sep 16 18:45:07.644528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.644425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"}
Sep 16 18:45:07.644528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.644452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"}
Sep 16 18:45:07.644528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.644477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"}
Sep 16 18:45:07.644528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.644501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"}
Sep 16 18:45:07.644528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.644514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.648513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.647972    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:45:07.648513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.648044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:45:07.648513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.647980    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:45:07.648513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.648187    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:45:07.648513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.648203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:45:07.648513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.647980    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:45:07.648890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.648716    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:45:07.648890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.648778    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-11f86icj7jk9t\""
Sep 16 18:45:07.648890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.648805    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:45:07.649706 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.649689    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:45:07.651154 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.651130    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:45:07.651717 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.651698    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:45:07.651799 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.651701    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mqkf\""
Sep 16 18:45:07.652052 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.652035    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:45:07.652216 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.652138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:45:07.652391 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.652374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:45:07.665106 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.665085    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:45:07.669791 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.669772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.692170 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692538 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnt6\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-kube-api-access-jhnt6\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692538 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692538 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692538 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-web-config\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692538 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.692839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.693214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.693214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692864    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.693214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.693214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.693214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.692978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.693214 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.693007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config-out\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793499 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793684 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793684 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793778 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793925 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793925 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.793925 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.794068 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.794068 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.793973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.794190 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.794158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.794374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.794353    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.794792    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.794923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config-out\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.794964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.794989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnt6\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-kube-api-access-jhnt6\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.795028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.795051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.795082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-web-config\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.795228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.795106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.797173 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.795111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.798401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.798349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.799142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.799394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.799501    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.799560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.799854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.799906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.800834 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.800546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801177 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.800978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801177 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.801002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config-out\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801177 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.801138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801388 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.801226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801388 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.801309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801491 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.801456    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-web-config\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.801594 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.801577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.804583 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.804531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnt6\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-kube-api-access-jhnt6\") pod \"prometheus-k8s-0\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:07.854577 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.854541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.943894 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.943798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.964775 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:07.964733    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:08.362841 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.362813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:45:08.365158 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:08.365129    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda727919e_4f78_4b6b_81b9_f5260e0c4eda.slice/crio-64680a5fd1fb1ad23ba19f06dc8a517cb7bd2c206689003daae936314ab01642 WatchSource:0}: Error finding container 64680a5fd1fb1ad23ba19f06dc8a517cb7bd2c206689003daae936314ab01642: Status 404 returned error can't find the container with id 64680a5fd1fb1ad23ba19f06dc8a517cb7bd2c206689003daae936314ab01642
Sep 16 18:45:08.647230 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.647149    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb" exitCode=0
Sep 16 18:45:08.647631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.647235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"}
Sep 16 18:45:08.647631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.647311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"64680a5fd1fb1ad23ba19f06dc8a517cb7bd2c206689003daae936314ab01642"}
Sep 16 18:45:08.650969 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.650947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerStarted","Data":"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"}
Sep 16 18:45:08.652519 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.652500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8" event={"ID":"fc57feab-c598-46a1-8d33-2ca94832f77f","Type":"ContainerStarted","Data":"3ffc22cbec9886cc610f2f6ee60c490733f4888bdfc182fea75ddccd12daba0b"}
Sep 16 18:45:08.699213 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.699160    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.632558285 podStartE2EDuration="6.699144604s" totalImagesPullingTime="3.050780459s" totalInitContainerRuntime="15.80586ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.696870438 +0000 UTC m=+111.113573235" watchObservedRunningTime="2026-09-16 18:45:08.699144604 +0000 UTC m=+111.115847362"
Sep 16 18:45:08.718737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:08.718690    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8" podStartSLOduration=2.357572889 podStartE2EDuration="3.718674236s" totalImagesPullingTime="1.361101347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:08.717675509 +0000 UTC m=+111.134378268" watchObservedRunningTime="2026-09-16 18:45:08.718674236 +0000 UTC m=+111.135376994"
Sep 16 18:45:11.343357 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:11.343303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.407779 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:11.407736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.599787 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:11.599712    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cfxqx"
Sep 16 18:45:12.671388 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:12.671356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"}
Sep 16 18:45:12.671725 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:12.671420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"}
Sep 16 18:45:13.540021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:13.539985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.651636 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:13.651598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.796203 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:13.796132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.862742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:13.862712    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b5bc4865-7zbhp"]
Sep 16 18:45:13.867492 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:13.867468    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:45:13.943646 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:13.943610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.068017 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.067984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.174167 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.174122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.269021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.268989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.563301 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.563274    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.664010 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.663982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.687518 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.687483    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"}
Sep 16 18:45:14.687681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.687571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"}
Sep 16 18:45:14.687681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.687597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"}
Sep 16 18:45:14.687681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.687622    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerStarted","Data":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"}
Sep 16 18:45:14.731519 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:14.731456    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.153942358 podStartE2EDuration="7.731417027s" totalImagesPullingTime="5.557836304s" totalInitContainerRuntime="19.638365ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:14.729155875 +0000 UTC m=+117.145858634" watchObservedRunningTime="2026-09-16 18:45:14.731417027 +0000 UTC m=+117.148119788"
Sep 16 18:45:15.042519 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:15.042429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.154877 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:15.154847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.245421 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:15.245392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.986565 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:15.986509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.536437 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:16.536405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:16.631016 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:16.630966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:17.965720 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:17.965687    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:18.010066 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:18.010036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:18.511390 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:18.511350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:18.615046 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:18.615013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:18.709795 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:18.709747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:23.036056 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:23.036021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:23.290062 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:23.289967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:26.139561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:26.139531    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:26.139961 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:26.139587    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:26.936579 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:26.936550    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:27.001309 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:27.001281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:27.717379 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:27.717299    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:27.990149 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:27.990054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:45:27.992575 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:27.992547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb5a23e-f4e8-4542-b175-5fe7be200604-metrics-certs\") pod \"network-metrics-daemon-5ps6t\" (UID: \"2cb5a23e-f4e8-4542-b175-5fe7be200604\") " pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:45:28.144582 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:28.144558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7bds\""
Sep 16 18:45:28.153057 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:28.153039    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ps6t"
Sep 16 18:45:28.425702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:28.425677    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ps6t"]
Sep 16 18:45:28.428081 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:45:28.428053    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb5a23e_f4e8_4542_b175_5fe7be200604.slice/crio-fbf93ee3df2361c1752f396f8d034913e242bd1560c7763bf95c6bf4013efe93 WatchSource:0}: Error finding container fbf93ee3df2361c1752f396f8d034913e242bd1560c7763bf95c6bf4013efe93: Status 404 returned error can't find the container with id fbf93ee3df2361c1752f396f8d034913e242bd1560c7763bf95c6bf4013efe93
Sep 16 18:45:28.747497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:28.747414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ps6t" event={"ID":"2cb5a23e-f4e8-4542-b175-5fe7be200604","Type":"ContainerStarted","Data":"fbf93ee3df2361c1752f396f8d034913e242bd1560c7763bf95c6bf4013efe93"}
Sep 16 18:45:28.957462 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:28.957433    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:29.058158 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:29.058114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:29.753231 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:29.753149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ps6t" event={"ID":"2cb5a23e-f4e8-4542-b175-5fe7be200604","Type":"ContainerStarted","Data":"697cf2badc78da82845d8ca6aa27f695ed1142878bc4143a77d1a9794fe7d4f2"}
Sep 16 18:45:29.753231 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:29.753210    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ps6t" event={"ID":"2cb5a23e-f4e8-4542-b175-5fe7be200604","Type":"ContainerStarted","Data":"beb164a6d11db0e5cacd5c93f0f10cf3761a6896ed0ec9ed7cc65163e97d90a0"}
Sep 16 18:45:29.770287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:29.770241    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5ps6t" podStartSLOduration=129.991022281 podStartE2EDuration="2m10.770226459s" totalImagesPullingTime="779.204178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:29.76928359 +0000 UTC m=+132.185986350" watchObservedRunningTime="2026-09-16 18:45:29.770226459 +0000 UTC m=+132.186929218"
Sep 16 18:45:29.867011 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:29.866983    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:29.935213 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:29.935180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:32.877411 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:32.877376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:32.931011 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:32.930976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:34.779305 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:34.779224    2562 generic.go:356] "Generic (PLEG): container finished" podID="a180ce87-cf24-4c64-bbc8-0dd2ae85013f" containerID="b15e05ea65dc1076d78687908d58811f3d36ae4ee2e173833595e7a2d4ac5a0f" exitCode=0
Sep 16 18:45:34.779305 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:34.779299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gn5r4" event={"ID":"a180ce87-cf24-4c64-bbc8-0dd2ae85013f","Type":"ContainerDied","Data":"b15e05ea65dc1076d78687908d58811f3d36ae4ee2e173833595e7a2d4ac5a0f"}
Sep 16 18:45:34.779703 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:34.779673    2562 scope.go:117] "RemoveContainer" containerID="b15e05ea65dc1076d78687908d58811f3d36ae4ee2e173833595e7a2d4ac5a0f"
Sep 16 18:45:35.043552 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:35.043468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.101034 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:35.101003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.785884 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:35.785846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gn5r4" event={"ID":"a180ce87-cf24-4c64-bbc8-0dd2ae85013f","Type":"ContainerStarted","Data":"1c44d49cb77d8ba6130b18218487280e90cb1afcac5c0067a844cd6b2d568019"}
Sep 16 18:45:36.857037 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:36.857002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:36.955500 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:36.955464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:37.055222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:37.055195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:38.884579 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:38.884538    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp" podUID="07599eaa-c564-4ecd-81d3-6bd4a70ebc75" containerName="registry" containerID="cri-o://47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d" gracePeriod=30
Sep 16 18:45:39.123410 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.123384    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:45:39.278887 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.278800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-278sf\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-kube-api-access-278sf\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.278887 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.278848    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-trusted-ca\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.278887 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.278870    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-certificates\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.279148 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.278903    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-installation-pull-secrets\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.279148 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.279148 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279099    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-bound-sa-token\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.279286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279162    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-image-registry-private-configuration\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.279286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279195    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-ca-trust-extracted\") pod \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\" (UID: \"07599eaa-c564-4ecd-81d3-6bd4a70ebc75\") "
Sep 16 18:45:39.279420 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279355    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:39.279475 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:39.279604 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279528    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-trusted-ca\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.279681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.279610    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-certificates\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.281811 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.281773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:39.281930 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.281876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:39.282055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.282025    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:39.282162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.282063    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:39.282206 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.282155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-kube-api-access-278sf" (OuterVolumeSpecName: "kube-api-access-278sf") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "kube-api-access-278sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:39.286968 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.286946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "07599eaa-c564-4ecd-81d3-6bd4a70ebc75" (UID: "07599eaa-c564-4ecd-81d3-6bd4a70ebc75"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:39.380567 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.380527    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278sf\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-kube-api-access-278sf\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.380567 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.380556    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-installation-pull-secrets\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.380567 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.380566    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-registry-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.380567 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.380574    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-bound-sa-token\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.380816 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.380585    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-image-registry-private-configuration\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.380816 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.380594    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/07599eaa-c564-4ecd-81d3-6bd4a70ebc75-ca-trust-extracted\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:45:39.803283 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.803249    2562 generic.go:356] "Generic (PLEG): container finished" podID="07599eaa-c564-4ecd-81d3-6bd4a70ebc75" containerID="47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d" exitCode=0
Sep 16 18:45:39.803449 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.803296    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp"
Sep 16 18:45:39.803449 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.803353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp" event={"ID":"07599eaa-c564-4ecd-81d3-6bd4a70ebc75","Type":"ContainerDied","Data":"47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d"}
Sep 16 18:45:39.803449 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.803408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b5bc4865-7zbhp" event={"ID":"07599eaa-c564-4ecd-81d3-6bd4a70ebc75","Type":"ContainerDied","Data":"dba485695294203eda55db3e21e543c6e8e854dce989d9682c74f0890fbeaaa5"}
Sep 16 18:45:39.803449 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.803434    2562 scope.go:117] "RemoveContainer" containerID="47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d"
Sep 16 18:45:39.815799 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.814013    2562 scope.go:117] "RemoveContainer" containerID="47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d"
Sep 16 18:45:39.815799 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:45:39.815473    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d\": container with ID starting with 47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d not found: ID does not exist" containerID="47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d"
Sep 16 18:45:39.815799 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.815607    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d"} err="failed to get container status \"47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d\": rpc error: code = NotFound desc = could not find container \"47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d\": container with ID starting with 47b44a325a6a34c529ddc0db3c32d3233b8b9a733f598c0803d652c53961813d not found: ID does not exist"
Sep 16 18:45:39.836805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.836775    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b5bc4865-7zbhp"]
Sep 16 18:45:39.838923 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:39.838897    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59b5bc4865-7zbhp"]
Sep 16 18:45:40.122410 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:40.122311    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07599eaa-c564-4ecd-81d3-6bd4a70ebc75" path="/var/lib/kubelet/pods/07599eaa-c564-4ecd-81d3-6bd4a70ebc75/volumes"
Sep 16 18:45:41.305994 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:41.305960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.378408 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:41.378373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.830041 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:45.830006    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec65898b-303d-405e-9830-5e9d0a05c616" containerID="3feeccf68022c805c17d7a02f12d22fcc6c55ff347099a33658266390329f380" exitCode=0
Sep 16 18:45:45.830573 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:45.830090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg" event={"ID":"ec65898b-303d-405e-9830-5e9d0a05c616","Type":"ContainerDied","Data":"3feeccf68022c805c17d7a02f12d22fcc6c55ff347099a33658266390329f380"}
Sep 16 18:45:45.830573 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:45.830542    2562 scope.go:117] "RemoveContainer" containerID="3feeccf68022c805c17d7a02f12d22fcc6c55ff347099a33658266390329f380"
Sep 16 18:45:46.144829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:46.144742    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:46.154332 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:46.154290    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bf57fd676-rx9x8"
Sep 16 18:45:46.837801 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:46.837749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-jtxxg" event={"ID":"ec65898b-303d-405e-9830-5e9d0a05c616","Type":"ContainerStarted","Data":"cd35be1fc327dc11bbef1bfeb025169ab73f58389cc1cda9d0ebdbd7e78506e9"}
Sep 16 18:45:56.303437 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:56.303401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.359088 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:45:56.359056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:07.965850 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:07.965813    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:07.985779 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:07.985753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:08.946575 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:08.946545    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:11.345045 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:11.344963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.650647 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:20.650608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:20.919257 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:20.919174    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.722977 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.722944    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:21.723365 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.723334    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="alertmanager" containerID="cri-o://4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3" gracePeriod=120
Sep 16 18:46:21.723425 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.723389    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="config-reloader" containerID="cri-o://08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157" gracePeriod=120
Sep 16 18:46:21.723425 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.723391    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy" containerID="cri-o://0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115" gracePeriod=120
Sep 16 18:46:21.723512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.723361    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-metric" containerID="cri-o://e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf" gracePeriod=120
Sep 16 18:46:21.723512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.723387    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-web" containerID="cri-o://8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf" gracePeriod=120
Sep 16 18:46:21.723512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.723413    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="prom-label-proxy" containerID="cri-o://36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5" gracePeriod=120
Sep 16 18:46:21.988742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988651    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5" exitCode=0
Sep 16 18:46:21.988742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988674    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115" exitCode=0
Sep 16 18:46:21.988742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988680    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157" exitCode=0
Sep 16 18:46:21.988742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988685    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3" exitCode=0
Sep 16 18:46:21.989019 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"}
Sep 16 18:46:21.989019 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"}
Sep 16 18:46:21.989019 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"}
Sep 16 18:46:21.989019 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:21.988897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"}
Sep 16 18:46:22.965119 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.965096    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:22.999006 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.998971    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf" exitCode=0
Sep 16 18:46:22.999006 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.999002    2562 generic.go:356] "Generic (PLEG): container finished" podID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerID="8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf" exitCode=0
Sep 16 18:46:22.999222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.999039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"}
Sep 16 18:46:22.999222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.999068    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:22.999222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.999113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"}
Sep 16 18:46:22.999222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.999154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"41f78a60-1e5e-484c-8dac-e7aa6b80de61","Type":"ContainerDied","Data":"8d8e5fa743a327f61ec90d4e142a964117bbd8b2e56cf38c83b89cdcb7bbf21d"}
Sep 16 18:46:22.999222 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:22.999190    2562 scope.go:117] "RemoveContainer" containerID="36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"
Sep 16 18:46:23.008427 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.008404    2562 scope.go:117] "RemoveContainer" containerID="e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"
Sep 16 18:46:23.015845 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.015813    2562 scope.go:117] "RemoveContainer" containerID="0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"
Sep 16 18:46:23.022749 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.022731    2562 scope.go:117] "RemoveContainer" containerID="8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"
Sep 16 18:46:23.029549 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.029533    2562 scope.go:117] "RemoveContainer" containerID="08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"
Sep 16 18:46:23.037518 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.037500    2562 scope.go:117] "RemoveContainer" containerID="4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"
Sep 16 18:46:23.044525 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.044508    2562 scope.go:117] "RemoveContainer" containerID="243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"
Sep 16 18:46:23.047705 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-web\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.047809 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-out\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.047809 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.047809 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047771    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-volume\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.047809 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-main-db\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8h9m\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-kube-api-access-j8h9m\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047872    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-metric\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047904    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-tls-assets\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048013 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.047997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048265 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-cluster-tls-config\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048265 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048097    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-web-config\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048265 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048143    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-metrics-client-ca\") pod \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\" (UID: \"41f78a60-1e5e-484c-8dac-e7aa6b80de61\") "
Sep 16 18:46:23.048265 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:23.048590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048429    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.049171 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048817    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:23.049171 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.048988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:23.051110 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.051077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-out" (OuterVolumeSpecName: "config-out") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:23.052025 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.051997    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.052504 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.052469    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:23.053282 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.053252    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.053609 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.053583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-volume" (OuterVolumeSpecName: "config-volume") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.053917 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.053882    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.054208 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.054186    2562 scope.go:117] "RemoveContainer" containerID="36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"
Sep 16 18:46:23.054584 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.054558    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5\": container with ID starting with 36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5 not found: ID does not exist" containerID="36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"
Sep 16 18:46:23.054675 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.054597    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"} err="failed to get container status \"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5\": rpc error: code = NotFound desc = could not find container \"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5\": container with ID starting with 36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5 not found: ID does not exist"
Sep 16 18:46:23.054675 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.054624    2562 scope.go:117] "RemoveContainer" containerID="e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"
Sep 16 18:46:23.054950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.054925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.055011 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.054931    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf\": container with ID starting with e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf not found: ID does not exist" containerID="e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"
Sep 16 18:46:23.055011 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.054980    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"} err="failed to get container status \"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf\": rpc error: code = NotFound desc = could not find container \"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf\": container with ID starting with e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf not found: ID does not exist"
Sep 16 18:46:23.055075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.055017    2562 scope.go:117] "RemoveContainer" containerID="0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"
Sep 16 18:46:23.055164 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.055141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-kube-api-access-j8h9m" (OuterVolumeSpecName: "kube-api-access-j8h9m") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "kube-api-access-j8h9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:23.055337 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.055301    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115\": container with ID starting with 0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115 not found: ID does not exist" containerID="0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"
Sep 16 18:46:23.055473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.055441    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"} err="failed to get container status \"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115\": rpc error: code = NotFound desc = could not find container \"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115\": container with ID starting with 0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115 not found: ID does not exist"
Sep 16 18:46:23.055473 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.055472    2562 scope.go:117] "RemoveContainer" containerID="8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"
Sep 16 18:46:23.055793 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.055773    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf\": container with ID starting with 8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf not found: ID does not exist" containerID="8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"
Sep 16 18:46:23.055855 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.055801    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"} err="failed to get container status \"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf\": rpc error: code = NotFound desc = could not find container \"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf\": container with ID starting with 8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf not found: ID does not exist"
Sep 16 18:46:23.055855 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.055816    2562 scope.go:117] "RemoveContainer" containerID="08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"
Sep 16 18:46:23.056099 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.056068    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157\": container with ID starting with 08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157 not found: ID does not exist" containerID="08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"
Sep 16 18:46:23.056161 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.056092    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"} err="failed to get container status \"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157\": rpc error: code = NotFound desc = could not find container \"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157\": container with ID starting with 08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157 not found: ID does not exist"
Sep 16 18:46:23.056161 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.056111    2562 scope.go:117] "RemoveContainer" containerID="4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"
Sep 16 18:46:23.056507 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.056486    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3\": container with ID starting with 4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3 not found: ID does not exist" containerID="4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"
Sep 16 18:46:23.056634 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.056511    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"} err="failed to get container status \"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3\": rpc error: code = NotFound desc = could not find container \"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3\": container with ID starting with 4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3 not found: ID does not exist"
Sep 16 18:46:23.056634 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.056530    2562 scope.go:117] "RemoveContainer" containerID="243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"
Sep 16 18:46:23.056877 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:23.056818    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb\": container with ID starting with 243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb not found: ID does not exist" containerID="243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"
Sep 16 18:46:23.056877 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.056855    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"} err="failed to get container status \"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb\": rpc error: code = NotFound desc = could not find container \"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb\": container with ID starting with 243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb not found: ID does not exist"
Sep 16 18:46:23.056877 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.056877    2562 scope.go:117] "RemoveContainer" containerID="36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"
Sep 16 18:46:23.057162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.057142    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5"} err="failed to get container status \"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5\": rpc error: code = NotFound desc = could not find container \"36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5\": container with ID starting with 36dd862c9fb20cd04cf4af3b288462d3e23022851e62fcd194229f28434afed5 not found: ID does not exist"
Sep 16 18:46:23.057162 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.057162    2562 scope.go:117] "RemoveContainer" containerID="e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"
Sep 16 18:46:23.057481 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.057460    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf"} err="failed to get container status \"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf\": rpc error: code = NotFound desc = could not find container \"e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf\": container with ID starting with e03d3a414f0bed086316deb7cd12bf9ff6df3a3be84a11b3ec0325bd52c74ccf not found: ID does not exist"
Sep 16 18:46:23.057571 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.057482    2562 scope.go:117] "RemoveContainer" containerID="0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"
Sep 16 18:46:23.057818 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.057745    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115"} err="failed to get container status \"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115\": rpc error: code = NotFound desc = could not find container \"0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115\": container with ID starting with 0f8a86aa16b221dc81f9c72d24c94945ddcb057b5d2a64a3b24ece7750942115 not found: ID does not exist"
Sep 16 18:46:23.057818 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.057767    2562 scope.go:117] "RemoveContainer" containerID="8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"
Sep 16 18:46:23.058100 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058001    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf"} err="failed to get container status \"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf\": rpc error: code = NotFound desc = could not find container \"8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf\": container with ID starting with 8d11288923d135df82c0eb66a2e9887b2a75ed2ebbcde16f9fc8ff8aac75d4bf not found: ID does not exist"
Sep 16 18:46:23.058100 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058021    2562 scope.go:117] "RemoveContainer" containerID="08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"
Sep 16 18:46:23.058261 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058244    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157"} err="failed to get container status \"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157\": rpc error: code = NotFound desc = could not find container \"08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157\": container with ID starting with 08fdd6d7bda842bb72631b96a9a135e1fcce02f5cbbe182fa829a7b4e55b0157 not found: ID does not exist"
Sep 16 18:46:23.058409 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058263    2562 scope.go:117] "RemoveContainer" containerID="4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"
Sep 16 18:46:23.058553 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058534    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3"} err="failed to get container status \"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3\": rpc error: code = NotFound desc = could not find container \"4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3\": container with ID starting with 4fa96233642d70d7c779f400a5b208eb5338dafcd23d19eef5a84201134fe8d3 not found: ID does not exist"
Sep 16 18:46:23.058601 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058556    2562 scope.go:117] "RemoveContainer" containerID="243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"
Sep 16 18:46:23.058972 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058840    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.058972 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.058843    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb"} err="failed to get container status \"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb\": rpc error: code = NotFound desc = could not find container \"243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb\": container with ID starting with 243093abe80fcf44b3da4082933be76b9fba2c9cdea47acda9a05f12d37ed2bb not found: ID does not exist"
Sep 16 18:46:23.064796 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.064777    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-web-config" (OuterVolumeSpecName: "web-config") pod "41f78a60-1e5e-484c-8dac-e7aa6b80de61" (UID: "41f78a60-1e5e-484c-8dac-e7aa6b80de61"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:23.149591 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149561    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-main-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149591 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149587    2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-cluster-tls-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149591 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149597    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-web-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149605    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41f78a60-1e5e-484c-8dac-e7aa6b80de61-metrics-client-ca\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149614    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149622    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-out\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149630    2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-config-volume\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149637    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/41f78a60-1e5e-484c-8dac-e7aa6b80de61-alertmanager-main-db\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149645    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8h9m\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-kube-api-access-j8h9m\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149654    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149662    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41f78a60-1e5e-484c-8dac-e7aa6b80de61-tls-assets\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.149784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.149671    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41f78a60-1e5e-484c-8dac-e7aa6b80de61-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:23.323391 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.323364    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:23.327966 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.327939    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:23.356599 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356573    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:23.356850 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356838    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy"
Sep 16 18:46:23.356890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356852    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy"
Sep 16 18:46:23.356890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356861    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="prom-label-proxy"
Sep 16 18:46:23.356890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356866    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="prom-label-proxy"
Sep 16 18:46:23.356890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356872    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-web"
Sep 16 18:46:23.356890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356879    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-web"
Sep 16 18:46:23.357032 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.356901    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="init-config-reloader"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="init-config-reloader"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357239    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07599eaa-c564-4ecd-81d3-6bd4a70ebc75" containerName="registry"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357249    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07599eaa-c564-4ecd-81d3-6bd4a70ebc75" containerName="registry"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357261    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="config-reloader"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357277    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="config-reloader"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357292    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="alertmanager"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357311    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="alertmanager"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357405    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-metric"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357418    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-metric"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357620    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="alertmanager"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357633    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357651    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-web"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357662    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="prom-label-proxy"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357678    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="config-reloader"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357690    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07599eaa-c564-4ecd-81d3-6bd4a70ebc75" containerName="registry"
Sep 16 18:46:23.360606 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.357705    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" containerName="kube-rbac-proxy-metric"
Sep 16 18:46:23.363336 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.363293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.365740 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.365717    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:46:23.365859 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.365739    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:46:23.365859 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.365765    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-skjk4\""
Sep 16 18:46:23.365990 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.365970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:46:23.366055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.366045    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:46:23.366109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.366087    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:46:23.366652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.366627    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:46:23.366652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.366638    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:46:23.366652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.366651    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:46:23.371512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.371497    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:46:23.374119 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.374098    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:23.452354 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e30a1f74-7125-4828-a88f-1ad184201b9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e30a1f74-7125-4828-a88f-1ad184201b9b-config-out\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452483 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452501    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e30a1f74-7125-4828-a88f-1ad184201b9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e30a1f74-7125-4828-a88f-1ad184201b9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-web-config\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsh8\" (UniqueName: \"kubernetes.io/projected/e30a1f74-7125-4828-a88f-1ad184201b9b-kube-api-access-pnsh8\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a1f74-7125-4828-a88f-1ad184201b9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.452678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.452648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553380 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e30a1f74-7125-4828-a88f-1ad184201b9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e30a1f74-7125-4828-a88f-1ad184201b9b-config-out\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553731 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e30a1f74-7125-4828-a88f-1ad184201b9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553731 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e30a1f74-7125-4828-a88f-1ad184201b9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553822 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e30a1f74-7125-4828-a88f-1ad184201b9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553822 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-web-config\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553984 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsh8\" (UniqueName: \"kubernetes.io/projected/e30a1f74-7125-4828-a88f-1ad184201b9b-kube-api-access-pnsh8\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553984 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553984 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a1f74-7125-4828-a88f-1ad184201b9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.553984 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.553956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.554462 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.554414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e30a1f74-7125-4828-a88f-1ad184201b9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.556357 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.556307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e30a1f74-7125-4828-a88f-1ad184201b9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557170 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.556901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557170 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.556947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557170 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.557086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e30a1f74-7125-4828-a88f-1ad184201b9b-config-out\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557170 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.557145    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557508 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.557487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-web-config\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557543 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.557528    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557590 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.557571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e30a1f74-7125-4828-a88f-1ad184201b9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.557625 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.557611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.558551 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.558537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e30a1f74-7125-4828-a88f-1ad184201b9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.562655 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.562636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsh8\" (UniqueName: \"kubernetes.io/projected/e30a1f74-7125-4828-a88f-1ad184201b9b-kube-api-access-pnsh8\") pod \"alertmanager-main-0\" (UID: \"e30a1f74-7125-4828-a88f-1ad184201b9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.673804 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.673775    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:23.950258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:23.950235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:23.952655 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:46:23.952627    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30a1f74_7125_4828_a88f_1ad184201b9b.slice/crio-95acbf11524031f572564432e3d2a7c487d2ef53a55121cebc1ebce7f2000165 WatchSource:0}: Error finding container 95acbf11524031f572564432e3d2a7c487d2ef53a55121cebc1ebce7f2000165: Status 404 returned error can't find the container with id 95acbf11524031f572564432e3d2a7c487d2ef53a55121cebc1ebce7f2000165
Sep 16 18:46:24.003616 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:24.003588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"95acbf11524031f572564432e3d2a7c487d2ef53a55121cebc1ebce7f2000165"}
Sep 16 18:46:24.121446 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:24.121414    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f78a60-1e5e-484c-8dac-e7aa6b80de61" path="/var/lib/kubelet/pods/41f78a60-1e5e-484c-8dac-e7aa6b80de61/volumes"
Sep 16 18:46:25.009528 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.009488    2562 generic.go:356] "Generic (PLEG): container finished" podID="e30a1f74-7125-4828-a88f-1ad184201b9b" containerID="e0830dca27c024ccf44ec9751d39a613d011e395ed21ebec26f802ec58615e10" exitCode=0
Sep 16 18:46:25.009882 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.009584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerDied","Data":"e0830dca27c024ccf44ec9751d39a613d011e395ed21ebec26f802ec58615e10"}
Sep 16 18:46:25.742868 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.742790    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"]
Sep 16 18:46:25.745491 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.745464    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.747890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.747867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:46:25.748035 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.748005    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:46:25.748263 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.748247    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:46:25.748434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.748373    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:46:25.748434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.748417    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s6g88\""
Sep 16 18:46:25.748561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.748509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:46:25.756472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.756310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:46:25.760385 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.760357    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"]
Sep 16 18:46:25.874787 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-metrics-client-ca\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.874950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-serving-certs-ca-bundle\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.874950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n5t\" (UniqueName: \"kubernetes.io/projected/e3bafb0d-b88a-484d-9500-e5514255974a-kube-api-access-k2n5t\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.874950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-secret-telemeter-client\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.874950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.874950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-federate-client-tls\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.874950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-telemeter-client-tls\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.875128 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.874969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975396 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-metrics-client-ca\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-serving-certs-ca-bundle\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975560 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n5t\" (UniqueName: \"kubernetes.io/projected/e3bafb0d-b88a-484d-9500-e5514255974a-kube-api-access-k2n5t\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975560 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-secret-telemeter-client\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975622 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975622 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-federate-client-tls\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.975715 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.975625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-telemeter-client-tls\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.976360 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.976338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-serving-certs-ca-bundle\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.976672 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.976428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.976672 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.976498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3bafb0d-b88a-484d-9500-e5514255974a-metrics-client-ca\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.978313 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.978288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-federate-client-tls\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.978413 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.978358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-secret-telemeter-client\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.978453 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.978435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.978545 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.978526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e3bafb0d-b88a-484d-9500-e5514255974a-telemeter-client-tls\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:25.983059 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:25.983039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n5t\" (UniqueName: \"kubernetes.io/projected/e3bafb0d-b88a-484d-9500-e5514255974a-kube-api-access-k2n5t\") pod \"telemeter-client-7fbfc7d557-jzvtg\" (UID: \"e3bafb0d-b88a-484d-9500-e5514255974a\") " pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:26.016942 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.016867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"a29aee174837b2e53b44fb7a8b6ed3888e988eb53d27adcdddac0a936a510342"}
Sep 16 18:46:26.016942 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.016925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"5ce687bc7441bcb61abb294acfcffe62c198c645321a86ddc27eefcd161dfac7"}
Sep 16 18:46:26.017346 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.016953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"0829359ac2ea9150232e1c5bdf79872be56b80f36706527a039de44f20be2986"}
Sep 16 18:46:26.017346 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.016977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"d06457d86bb90f8285b6c82ce905d3af8684eccd8b9ec9126ec1df0e7a7780ae"}
Sep 16 18:46:26.017346 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.017001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"dd44bf101dcc20b6702eba553721ca57ca6b76ed0dcf2cfd3a5a385d7f0ffdbd"}
Sep 16 18:46:26.017346 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.017022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e30a1f74-7125-4828-a88f-1ad184201b9b","Type":"ContainerStarted","Data":"846a4f65a08e524c360119c7331ade77d791ab217394ff60c297ac7d7dce4b0d"}
Sep 16 18:46:26.042627 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042601    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:26.043651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042902    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="prometheus" containerID="cri-o://84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7" gracePeriod=600
Sep 16 18:46:26.043651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042929    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="thanos-sidecar" containerID="cri-o://2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9" gracePeriod=600
Sep 16 18:46:26.043651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042946    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy" containerID="cri-o://010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85" gracePeriod=600
Sep 16 18:46:26.043651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042987    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404" gracePeriod=600
Sep 16 18:46:26.043651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042946    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-web" containerID="cri-o://63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7" gracePeriod=600
Sep 16 18:46:26.043651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.042951    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="config-reloader" containerID="cri-o://5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396" gracePeriod=600
Sep 16 18:46:26.046364 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.046294    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.02752189 podStartE2EDuration="3.046272469s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.750579ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:26.043368431 +0000 UTC m=+188.460071193" watchObservedRunningTime="2026-09-16 18:46:26.046272469 +0000 UTC m=+188.462975228"
Sep 16 18:46:26.057542 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.057481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"
Sep 16 18:46:26.287714 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.287686    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:26.342433 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.342405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg"]
Sep 16 18:46:26.344175 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:46:26.344144    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bafb0d_b88a_484d_9500_e5514255974a.slice/crio-8ff3603abdf3e9585a726a51e77ab86ce50265b668305277e17ec44f9ea4caa7 WatchSource:0}: Error finding container 8ff3603abdf3e9585a726a51e77ab86ce50265b668305277e17ec44f9ea4caa7: Status 404 returned error can't find the container with id 8ff3603abdf3e9585a726a51e77ab86ce50265b668305277e17ec44f9ea4caa7
Sep 16 18:46:26.378550 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378519    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-kube-rbac-proxy\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378717 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378576    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-grpc-tls\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378717 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378613    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-1\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378819 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-metrics-client-certs\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378819 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-tls-assets\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378921 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-db\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378921 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-tls\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.378921 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378908    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-web-config\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378934    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-2\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378975    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.379065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.378994    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-kubelet-serving-ca-bundle\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-serving-certs-ca-bundle\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379065 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-trusted-ca-bundle\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379089    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-thanos-prometheus-http-client-file\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379126    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-0\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379158    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379190    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhnt6\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-kube-api-access-jhnt6\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379217    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-metrics-client-ca\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379286    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379347    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config-out\") pod \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\" (UID: \"a727919e-4f78-4b6b-81b9-f5260e0c4eda\") "
Sep 16 18:46:26.379670 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.379635    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.380986 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.380137    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:26.380986 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.380517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.380986 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.380777    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.381645 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.381618    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.381762 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.381739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.382056 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.382001    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.382612 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.382576    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:26.383172 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383139    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.383455 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.383529 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383462    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.383529 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.383766 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config" (OuterVolumeSpecName: "config") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.383766 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383752    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config-out" (OuterVolumeSpecName: "config-out") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:26.383913 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.383878    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-kube-api-access-jhnt6" (OuterVolumeSpecName: "kube-api-access-jhnt6") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "kube-api-access-jhnt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:26.384107 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.384088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:26.384195 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.384163    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.384748 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.384729    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.384857 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.384840    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.394129 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.394106    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-web-config" (OuterVolumeSpecName: "web-config") pod "a727919e-4f78-4b6b-81b9-f5260e0c4eda" (UID: "a727919e-4f78-4b6b-81b9-f5260e0c4eda"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:26.480991 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.480956    2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.480991 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.480983    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-config-out\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.480991 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.480994    2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-kube-rbac-proxy\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481005    2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-grpc-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481014    2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-metrics-client-certs\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481022    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-tls-assets\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481030    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-db\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481038    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481046    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-web-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481054    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481062    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481070    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481079    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481087    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481097    2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481106    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481114    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a727919e-4f78-4b6b-81b9-f5260e0c4eda-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481123    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnt6\" (UniqueName: \"kubernetes.io/projected/a727919e-4f78-4b6b-81b9-f5260e0c4eda-kube-api-access-jhnt6\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:26.481210 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:26.481133    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a727919e-4f78-4b6b-81b9-f5260e0c4eda-configmap-metrics-client-ca\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:46:27.024401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024371    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404" exitCode=0
Sep 16 18:46:27.024401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024395    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85" exitCode=0
Sep 16 18:46:27.024401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024402    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7" exitCode=0
Sep 16 18:46:27.024401 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024408    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9" exitCode=0
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024413    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396" exitCode=0
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024418    2562 generic.go:356] "Generic (PLEG): container finished" podID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7" exitCode=0
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024453    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"}
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"}
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"}
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"}
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024609    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"}
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"}
Sep 16 18:46:27.024902 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.024706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a727919e-4f78-4b6b-81b9-f5260e0c4eda","Type":"ContainerDied","Data":"64680a5fd1fb1ad23ba19f06dc8a517cb7bd2c206689003daae936314ab01642"}
Sep 16 18:46:27.025915 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.025892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg" event={"ID":"e3bafb0d-b88a-484d-9500-e5514255974a","Type":"ContainerStarted","Data":"8ff3603abdf3e9585a726a51e77ab86ce50265b668305277e17ec44f9ea4caa7"}
Sep 16 18:46:27.033405 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.033284    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.040653 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.040638    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.047678 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.047516    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.049763 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.049742    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:27.053344 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.053307    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:27.055113 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.055097    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.061937 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.061922    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.069173 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.069153    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.076033 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076014    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.076329 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.076298    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.076380 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076345    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"} err="failed to get container status \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist"
Sep 16 18:46:27.076380 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076364    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.076638 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.076620    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.076675 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076644    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"} err="failed to get container status \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist"
Sep 16 18:46:27.076675 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076660    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.076918 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.076900    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.076953 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076923    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"} err="failed to get container status \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist"
Sep 16 18:46:27.076953 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.076939    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.077184 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.077170    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.077228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077189    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"} err="failed to get container status \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist"
Sep 16 18:46:27.077228 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077203    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.077460 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.077444    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.077504 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077464    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"} err="failed to get container status \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist"
Sep 16 18:46:27.077504 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077477    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.077717 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.077700    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.077760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077721    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"} err="failed to get container status \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist"
Sep 16 18:46:27.077760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077734    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.077964 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:46:27.077950    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.078016 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077966    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"} err="failed to get container status \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist"
Sep 16 18:46:27.078016 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.077977    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.078250 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.078231    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"} err="failed to get container status \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist"
Sep 16 18:46:27.078250 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.078250    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.078737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.078697    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"} err="failed to get container status \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist"
Sep 16 18:46:27.078886 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.078791    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.080286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.080258    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"} err="failed to get container status \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist"
Sep 16 18:46:27.080286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.080286    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.080654 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.080620    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"} err="failed to get container status \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist"
Sep 16 18:46:27.080729 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.080657    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.080962 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.080944    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"} err="failed to get container status \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist"
Sep 16 18:46:27.081024 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.080963    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.081254 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081235    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"} err="failed to get container status \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist"
Sep 16 18:46:27.081307 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081255    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.081531 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081514    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"} err="failed to get container status \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist"
Sep 16 18:46:27.081589 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081531    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.081745 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081728    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:27.081797 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081776    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"} err="failed to get container status \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist"
Sep 16 18:46:27.081797 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.081791    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.082036 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082019    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"} err="failed to get container status \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082034    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="init-config-reloader"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082037    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082049    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="init-config-reloader"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082061    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="thanos-sidecar"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082067    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="thanos-sidecar"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082079    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="config-reloader"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082085    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="config-reloader"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082091    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-web"
Sep 16 18:46:27.082095 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082097    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-web"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082103    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-thanos"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082108    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-thanos"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082120    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082129    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082136    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="prometheus"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082143    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="prometheus"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082203    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="thanos-sidecar"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082214    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="prometheus"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082220    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082227    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="config-reloader"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082232    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-web"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082239    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" containerName="kube-rbac-proxy-thanos"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082287    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"} err="failed to get container status \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist"
Sep 16 18:46:27.082472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082299    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.083055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082588    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"} err="failed to get container status \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist"
Sep 16 18:46:27.083055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082626    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.083055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082925    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"} err="failed to get container status \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist"
Sep 16 18:46:27.083055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.082946    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.083206 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083182    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"} err="failed to get container status \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist"
Sep 16 18:46:27.083258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083205    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.083440 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083422    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"} err="failed to get container status \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist"
Sep 16 18:46:27.083497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083441    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.083663 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083643    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"} err="failed to get container status \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist"
Sep 16 18:46:27.083710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083663    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.083897 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083878    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"} err="failed to get container status \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist"
Sep 16 18:46:27.083945 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.083898    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.084137 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084120    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"} err="failed to get container status \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist"
Sep 16 18:46:27.084183 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084138    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.084463 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084443    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"} err="failed to get container status \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist"
Sep 16 18:46:27.084463 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084462    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.084702 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084682    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"} err="failed to get container status \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist"
Sep 16 18:46:27.084753 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084702    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.084942 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084926    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"} err="failed to get container status \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist"
Sep 16 18:46:27.084978 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.084943    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.085154 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085138    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"} err="failed to get container status \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist"
Sep 16 18:46:27.085199 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085155    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.085421 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085403    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"} err="failed to get container status \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist"
Sep 16 18:46:27.085478 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085421    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.085647 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085629    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"} err="failed to get container status \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist"
Sep 16 18:46:27.085692 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085648    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.085888 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085871    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"} err="failed to get container status \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist"
Sep 16 18:46:27.085939 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.085888    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.086123 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086107    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"} err="failed to get container status \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist"
Sep 16 18:46:27.086182 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086123    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.086369 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086353    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"} err="failed to get container status \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist"
Sep 16 18:46:27.086407 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086370    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.086457 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086445    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.086572 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086553    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"} err="failed to get container status \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist"
Sep 16 18:46:27.086614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.086576    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.087055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087034    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"} err="failed to get container status \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist"
Sep 16 18:46:27.087055 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087051    2562 scope.go:117] "RemoveContainer" containerID="fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"
Sep 16 18:46:27.087313 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087295    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404"} err="failed to get container status \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": rpc error: code = NotFound desc = could not find container \"fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404\": container with ID starting with fc9d0a30189017a24c89c51e9bfc27c90eb16d8c26b4e0968d0f9585a9205404 not found: ID does not exist"
Sep 16 18:46:27.087400 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087314    2562 scope.go:117] "RemoveContainer" containerID="010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"
Sep 16 18:46:27.087578 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087558    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85"} err="failed to get container status \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": rpc error: code = NotFound desc = could not find container \"010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85\": container with ID starting with 010270f552ca859f136d959e4069b32416be6d63be237231ecdfa7ab72883f85 not found: ID does not exist"
Sep 16 18:46:27.087649 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087580    2562 scope.go:117] "RemoveContainer" containerID="63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"
Sep 16 18:46:27.087822 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087800    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7"} err="failed to get container status \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": rpc error: code = NotFound desc = could not find container \"63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7\": container with ID starting with 63d65f06dca25ea4818d3f1e94218cb88a42a796cd24b5746f30bb7490f067f7 not found: ID does not exist"
Sep 16 18:46:27.087873 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.087824    2562 scope.go:117] "RemoveContainer" containerID="2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"
Sep 16 18:46:27.088062 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088045    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9"} err="failed to get container status \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": rpc error: code = NotFound desc = could not find container \"2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9\": container with ID starting with 2ee7daf2e30fbb6e2a6d5cd47c48753b5a272f757d5834407eeab99de29481a9 not found: ID does not exist"
Sep 16 18:46:27.088132 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088063    2562 scope.go:117] "RemoveContainer" containerID="5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"
Sep 16 18:46:27.088295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088279    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396"} err="failed to get container status \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": rpc error: code = NotFound desc = could not find container \"5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396\": container with ID starting with 5efa1f18b9b25be3302eff498619909430809efee13aeb9bc88cb099bbbe6396 not found: ID does not exist"
Sep 16 18:46:27.088388 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088296    2562 scope.go:117] "RemoveContainer" containerID="84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"
Sep 16 18:46:27.088615 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088582    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7"} err="failed to get container status \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": rpc error: code = NotFound desc = could not find container \"84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7\": container with ID starting with 84e95815b3a5c487656fd6040bdca08839d8db8dd27e62d3f0cd4ca74a0d94a7 not found: ID does not exist"
Sep 16 18:46:27.088615 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088609    2562 scope.go:117] "RemoveContainer" containerID="1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"
Sep 16 18:46:27.088738 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088671    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:46:27.088928 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088912    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:46:27.088928 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088918    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb"} err="failed to get container status \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": rpc error: code = NotFound desc = could not find container \"1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb\": container with ID starting with 1fa5daf64e30cacccb77323cbe39f24a2bc5de236638a624cb1c3e11e3f92abb not found: ID does not exist"
Sep 16 18:46:27.088928 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.088960    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:46:27.089242 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:46:27.089564 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089397    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:46:27.089564 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089397    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-11f86icj7jk9t\""
Sep 16 18:46:27.089564 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089432    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mqkf\""
Sep 16 18:46:27.089564 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089401    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:46:27.089564 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089401    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:46:27.089842 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089644    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:46:27.089842 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089707    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:46:27.089842 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089816    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:46:27.089964 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:46:27.089964 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.089922    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:46:27.094823 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.094255    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:46:27.099824 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.099719    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:46:27.101928 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.101909    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:27.186335 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-config\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186846 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.186846 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186827    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwn5r\" (UniqueName: \"kubernetes.io/projected/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-kube-api-access-qwn5r\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-web-config\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.186978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-config-out\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187557 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.187557 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.187348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288276 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288477 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288477 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288588 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288588 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-config\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288690 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288741 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288793 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288793 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwn5r\" (UniqueName: \"kubernetes.io/projected/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-kube-api-access-qwn5r\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288793 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-web-config\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288921    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-config-out\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.288950 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289192 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.288975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289244 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289338 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289918 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289752    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289918 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289918 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289829    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289918 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.289918 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.290220 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.289925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.290220 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.290137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.291088 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.290803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.291955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.291925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.292298 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.292244    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.292648 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.292620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.292739 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.292703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-config\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.293174 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.293097    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.293395 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.293371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.293701 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.293627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.294704 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.294662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.295159 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.295111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.295641 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.295598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-config-out\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.295717 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.295639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.295820 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.295796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-web-config\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.297282 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.297261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.300591 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.300541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwn5r\" (UniqueName: \"kubernetes.io/projected/bac1f1e1-c420-4f63-a353-5060a5ccbcbc-kube-api-access-qwn5r\") pod \"prometheus-k8s-0\" (UID: \"bac1f1e1-c420-4f63-a353-5060a5ccbcbc\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.399338 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.399286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:27.685888 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:27.685796    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:27.690524 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:46:27.690492    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac1f1e1_c420_4f63_a353_5060a5ccbcbc.slice/crio-9280a1f8a2df4051d37da171f815b4748d64850d6310646b740c0a71d8d21710 WatchSource:0}: Error finding container 9280a1f8a2df4051d37da171f815b4748d64850d6310646b740c0a71d8d21710: Status 404 returned error can't find the container with id 9280a1f8a2df4051d37da171f815b4748d64850d6310646b740c0a71d8d21710
Sep 16 18:46:28.034760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:28.034677    2562 generic.go:356] "Generic (PLEG): container finished" podID="bac1f1e1-c420-4f63-a353-5060a5ccbcbc" containerID="23fdb98c6c2df28ea983f62d284ebe02244c91bd6a4370d738270279eaf8e10b" exitCode=0
Sep 16 18:46:28.034760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:28.034724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerDied","Data":"23fdb98c6c2df28ea983f62d284ebe02244c91bd6a4370d738270279eaf8e10b"}
Sep 16 18:46:28.035169 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:28.034799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"9280a1f8a2df4051d37da171f815b4748d64850d6310646b740c0a71d8d21710"}
Sep 16 18:46:28.124080 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:28.124040    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a727919e-4f78-4b6b-81b9-f5260e0c4eda" path="/var/lib/kubelet/pods/a727919e-4f78-4b6b-81b9-f5260e0c4eda/volumes"
Sep 16 18:46:29.042806 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.042707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"69052da79780c0b832c0e76e6e7f17f3ce60395c5b85f5d0dc517d98cf7617a8"}
Sep 16 18:46:29.043239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.042837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"3d38544ceb674b7ac5c4da319bdae33b448c84b6fb735fdef8b215791d9916fe"}
Sep 16 18:46:29.043239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.042882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"f9524bbdd36179b46741ec26ffff7718fe7f9d816764d7183349190344518eb3"}
Sep 16 18:46:29.043239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.042921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"dae1eb139e0b24b21da9bc9b1155e15373c0f1e5f9fdbea27c86d56f9329c224"}
Sep 16 18:46:29.043239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.042964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"058326ce4864127b2f00c7f628e9ceb4688c40cdd8dbed85ca7c6de0341c67c4"}
Sep 16 18:46:29.043239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.043023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bac1f1e1-c420-4f63-a353-5060a5ccbcbc","Type":"ContainerStarted","Data":"c9bc723f58659815646f87ed1e4006c5499e7effb564d3257c0f2620221fdf07"}
Sep 16 18:46:29.044790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.044764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg" event={"ID":"e3bafb0d-b88a-484d-9500-e5514255974a","Type":"ContainerStarted","Data":"b490cacb356a0e88b4d8ff5371404382c903e1fa934f1ef55560952b19e6e4bf"}
Sep 16 18:46:29.044901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.044809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg" event={"ID":"e3bafb0d-b88a-484d-9500-e5514255974a","Type":"ContainerStarted","Data":"419db70f09c73d5bd6376df5785cfe9fa87f44cd9eda2f0565e4661d4f978f7c"}
Sep 16 18:46:29.044901 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.044827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg" event={"ID":"e3bafb0d-b88a-484d-9500-e5514255974a","Type":"ContainerStarted","Data":"a619bf27b10a2b72387bf84af8fddda5d675c04a0a6f6ef29f2b165a31995e0a"}
Sep 16 18:46:29.070962 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.070907    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.049173227 podStartE2EDuration="2.070890857s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.71763ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:29.069728544 +0000 UTC m=+191.486431338" watchObservedRunningTime="2026-09-16 18:46:29.070890857 +0000 UTC m=+191.487593617"
Sep 16 18:46:29.092114 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.092044    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fbfc7d557-jzvtg" podStartSLOduration=2.135338848 podStartE2EDuration="4.092028195s" totalImagesPullingTime="1.956689347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:29.090602571 +0000 UTC m=+191.507305358" watchObservedRunningTime="2026-09-16 18:46:29.092028195 +0000 UTC m=+191.508730955"
Sep 16 18:46:29.219273 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.219245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.285912 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:29.285879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.399780 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:32.399745    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:40.393853 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:40.393822    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.447309 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:40.447279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.498741 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:46:40.498715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.843334 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:05.843271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:05.943124 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:05.943088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:27.399789 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:27.399749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:27.416178 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:27.416154    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:28.308734 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:28.308709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:35.950459 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:35.950408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:36.004573 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:36.004546    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.110109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:40.110031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.171091 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:40.171057    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.793070 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:47.793019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.850743 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:47.850709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:50.662298 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:50.662259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:50.770787 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:50.770753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:50.830990 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:50.830964    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:56.319498 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:56.319462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:56.420988 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:47:56.420936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:18.029855 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:48:18.029827    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:48:53.232441 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:48:53.232405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.674539 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:48:54.674506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:15.380661 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:15.380578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:15.480436 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:15.480404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:20.906157 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:20.906122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.195541 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:21.195462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.391084 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:21.391055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.541250 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:21.541164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.764748 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:21.764718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.245408 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.245376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.343452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.343420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.448152 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.448122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.544002 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.543920    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.674604 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.674547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.772113 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.772079    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.998172 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:22.998143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:23.098271 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:23.098244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:51.982810 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:49:51.982768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:13.928527 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:13.928495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:13.982960 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:13.982931    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:17.607911 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:17.607876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:24.285630 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:24.285599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:24.341764 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:24.341735    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:35.333014 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:35.332977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:35.386769 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:35.386735    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:35.440582 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:35.440549    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.516380 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:38.516354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.594305 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:38.594269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.683454 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:38.683420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.784893 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:38.784819    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:38.886042 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:38.886016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.342666 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:39.342638    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.443534 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:39.443502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.505589 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:50:39.505563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:36.135123 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:36.135090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:46.878184 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:46.878139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:55.219597 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:55.219554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:55.346498 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:55.346469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:55.603743 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:55.603711    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:55.871045 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:55.870966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:56.679120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:56.679086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:56.778773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:51:56.778742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:12.444949 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.444879    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cgr4p"]
Sep 16 18:52:12.448041 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.448025    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.453245 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.453220    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:52:12.453245 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.453236    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:52:12.453445 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.453312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:52:12.453514 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.453502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:52:12.454444 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.454096    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 18:52:12.459429 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.459397    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cgr4p"]
Sep 16 18:52:12.602756 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.602727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97xw\" (UniqueName: \"kubernetes.io/projected/152dcbb0-4b84-4852-8102-c09114d9fe39-kube-api-access-m97xw\") pod \"s3-init-cgr4p\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") " pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.602939 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.602766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152dcbb0-4b84-4852-8102-c09114d9fe39-model-cache\") pod \"s3-init-cgr4p\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") " pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.704059 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.703986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97xw\" (UniqueName: \"kubernetes.io/projected/152dcbb0-4b84-4852-8102-c09114d9fe39-kube-api-access-m97xw\") pod \"s3-init-cgr4p\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") " pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.704059 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.704021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152dcbb0-4b84-4852-8102-c09114d9fe39-model-cache\") pod \"s3-init-cgr4p\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") " pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.704395 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.704381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152dcbb0-4b84-4852-8102-c09114d9fe39-model-cache\") pod \"s3-init-cgr4p\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") " pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.713241 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.713222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97xw\" (UniqueName: \"kubernetes.io/projected/152dcbb0-4b84-4852-8102-c09114d9fe39-kube-api-access-m97xw\") pod \"s3-init-cgr4p\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") " pod="kserve/s3-init-cgr4p"
Sep 16 18:52:12.757238 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:12.757207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cgr4p"
Sep 16 18:52:13.027211 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:13.027166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cgr4p"]
Sep 16 18:52:13.033664 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:13.033642    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:13.408070 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:13.408040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cgr4p" event={"ID":"152dcbb0-4b84-4852-8102-c09114d9fe39","Type":"ContainerStarted","Data":"6e4ade7cea5c47b06a15056a7a0900017ac0a990cf6ad862aae15bb57b38f591"}
Sep 16 18:52:16.424403 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:16.424372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cgr4p" event={"ID":"152dcbb0-4b84-4852-8102-c09114d9fe39","Type":"ContainerStarted","Data":"c46d114623b2af180cd0f4d4c3a8e695500f5f7adfb79726ce25ec563380db4a"}
Sep 16 18:52:18.434802 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:18.434770    2562 generic.go:356] "Generic (PLEG): container finished" podID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerID="c46d114623b2af180cd0f4d4c3a8e695500f5f7adfb79726ce25ec563380db4a" exitCode=0
Sep 16 18:52:18.435188 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:18.434811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cgr4p" event={"ID":"152dcbb0-4b84-4852-8102-c09114d9fe39","Type":"ContainerDied","Data":"c46d114623b2af180cd0f4d4c3a8e695500f5f7adfb79726ce25ec563380db4a"}
Sep 16 18:52:22.833397 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:22.833377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:52:23.459839 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:23.459802    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cgr4p" event={"ID":"152dcbb0-4b84-4852-8102-c09114d9fe39","Type":"ContainerStarted","Data":"253e8231b47561ca4aed2c4150f73fe5275b82f4e5b3a48ffbc34c73c19bb847"}
Sep 16 18:52:23.479927 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:52:23.479873    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cgr4p" podStartSLOduration=2.323788588 podStartE2EDuration="11.479855915s" totalImagesPullingTime="7.688183315s" totalInitContainerRuntime="1.467884012s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:23.478892046 +0000 UTC m=+545.895594806" watchObservedRunningTime="2026-09-16 18:52:23.479855915 +0000 UTC m=+545.896558674"
Sep 16 18:53:02.418988 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:02.418956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:02.520564 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:02.520531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:28.720942 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:28.720911    2562 generic.go:356] "Generic (PLEG): container finished" podID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerID="253e8231b47561ca4aed2c4150f73fe5275b82f4e5b3a48ffbc34c73c19bb847" exitCode=0
Sep 16 18:53:28.721314 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:28.720992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cgr4p" event={"ID":"152dcbb0-4b84-4852-8102-c09114d9fe39","Type":"ContainerDied","Data":"253e8231b47561ca4aed2c4150f73fe5275b82f4e5b3a48ffbc34c73c19bb847"}
Sep 16 18:53:29.846041 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:29.846019    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cgr4p"
Sep 16 18:53:29.968869 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:29.968840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m97xw\" (UniqueName: \"kubernetes.io/projected/152dcbb0-4b84-4852-8102-c09114d9fe39-kube-api-access-m97xw\") pod \"152dcbb0-4b84-4852-8102-c09114d9fe39\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") "
Sep 16 18:53:29.969038 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:29.968882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152dcbb0-4b84-4852-8102-c09114d9fe39-model-cache\") pod \"152dcbb0-4b84-4852-8102-c09114d9fe39\" (UID: \"152dcbb0-4b84-4852-8102-c09114d9fe39\") "
Sep 16 18:53:29.971206 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:29.971174    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152dcbb0-4b84-4852-8102-c09114d9fe39-kube-api-access-m97xw" (OuterVolumeSpecName: "kube-api-access-m97xw") pod "152dcbb0-4b84-4852-8102-c09114d9fe39" (UID: "152dcbb0-4b84-4852-8102-c09114d9fe39"). InnerVolumeSpecName "kube-api-access-m97xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:29.988152 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:29.988121    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152dcbb0-4b84-4852-8102-c09114d9fe39-model-cache" (OuterVolumeSpecName: "model-cache") pod "152dcbb0-4b84-4852-8102-c09114d9fe39" (UID: "152dcbb0-4b84-4852-8102-c09114d9fe39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:30.070174 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:30.070139    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97xw\" (UniqueName: \"kubernetes.io/projected/152dcbb0-4b84-4852-8102-c09114d9fe39-kube-api-access-m97xw\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:53:30.070174 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:30.070169    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152dcbb0-4b84-4852-8102-c09114d9fe39-model-cache\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:53:30.731627 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:30.731597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cgr4p" event={"ID":"152dcbb0-4b84-4852-8102-c09114d9fe39","Type":"ContainerDied","Data":"6e4ade7cea5c47b06a15056a7a0900017ac0a990cf6ad862aae15bb57b38f591"}
Sep 16 18:53:30.731627 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:30.731610    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cgr4p"
Sep 16 18:53:30.731627 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:30.731633    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e4ade7cea5c47b06a15056a7a0900017ac0a990cf6ad862aae15bb57b38f591"
Sep 16 18:53:31.449741 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.449710    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"]
Sep 16 18:53:31.450098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.450040    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerName="s3-init"
Sep 16 18:53:31.450098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.450055    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerName="s3-init"
Sep 16 18:53:31.450098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.450068    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerName="download-hf-model"
Sep 16 18:53:31.450098 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.450076    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerName="download-hf-model"
Sep 16 18:53:31.450235 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.450175    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="152dcbb0-4b84-4852-8102-c09114d9fe39" containerName="s3-init"
Sep 16 18:53:31.453365 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.453337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.455626 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.455604    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:31.455757 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.455604    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:53:31.455912 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.455895    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 18:53:31.456434 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.456417    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:31.457028 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.456971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:31.461021 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.460999    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"]
Sep 16 18:53:31.584480 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.584445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/898072c6-66e4-4d46-9db9-5fa66f571154-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-qt8p6\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.584638 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.584488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kk5\" (UniqueName: \"kubernetes.io/projected/898072c6-66e4-4d46-9db9-5fa66f571154-kube-api-access-25kk5\") pod \"seaweedfs-tls-custom-6644bf5d6c-qt8p6\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.685239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.685204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/898072c6-66e4-4d46-9db9-5fa66f571154-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-qt8p6\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.685239 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.685242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kk5\" (UniqueName: \"kubernetes.io/projected/898072c6-66e4-4d46-9db9-5fa66f571154-kube-api-access-25kk5\") pod \"seaweedfs-tls-custom-6644bf5d6c-qt8p6\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.685611 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.685590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/898072c6-66e4-4d46-9db9-5fa66f571154-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-qt8p6\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.693733 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.693714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kk5\" (UniqueName: \"kubernetes.io/projected/898072c6-66e4-4d46-9db9-5fa66f571154-kube-api-access-25kk5\") pod \"seaweedfs-tls-custom-6644bf5d6c-qt8p6\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:31.763289 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:31.763222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:53:32.031602 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:32.031577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"]
Sep 16 18:53:32.033867 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:53:32.033837    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898072c6_66e4_4d46_9db9_5fa66f571154.slice/crio-888f435c5ffc0d99a389a46d35b5a1b15652f918c1b59c49cdfe9e3f370ab423 WatchSource:0}: Error finding container 888f435c5ffc0d99a389a46d35b5a1b15652f918c1b59c49cdfe9e3f370ab423: Status 404 returned error can't find the container with id 888f435c5ffc0d99a389a46d35b5a1b15652f918c1b59c49cdfe9e3f370ab423
Sep 16 18:53:32.742224 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:32.742192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6" event={"ID":"898072c6-66e4-4d46-9db9-5fa66f571154","Type":"ContainerStarted","Data":"888f435c5ffc0d99a389a46d35b5a1b15652f918c1b59c49cdfe9e3f370ab423"}
Sep 16 18:53:35.757296 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:35.757265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6" event={"ID":"898072c6-66e4-4d46-9db9-5fa66f571154","Type":"ContainerStarted","Data":"f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f"}
Sep 16 18:53:35.772425 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:35.772365    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6" podStartSLOduration=2.025811966 podStartE2EDuration="4.772305424s" totalImagesPullingTime="2.746493458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:35.771879718 +0000 UTC m=+618.188582477" watchObservedRunningTime="2026-09-16 18:53:35.772305424 +0000 UTC m=+618.189008183"
Sep 16 18:53:37.278507 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:37.278472    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"]
Sep 16 18:53:37.766204 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:53:37.766158    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6" podUID="898072c6-66e4-4d46-9db9-5fa66f571154" containerName="seaweedfs-tls-custom" containerID="cri-o://f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f" gracePeriod=30
Sep 16 18:54:05.501643 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.501617    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:54:05.570002 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.569976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25kk5\" (UniqueName: \"kubernetes.io/projected/898072c6-66e4-4d46-9db9-5fa66f571154-kube-api-access-25kk5\") pod \"898072c6-66e4-4d46-9db9-5fa66f571154\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") "
Sep 16 18:54:05.570169 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.570019    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/898072c6-66e4-4d46-9db9-5fa66f571154-data\") pod \"898072c6-66e4-4d46-9db9-5fa66f571154\" (UID: \"898072c6-66e4-4d46-9db9-5fa66f571154\") "
Sep 16 18:54:05.571883 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.571848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898072c6-66e4-4d46-9db9-5fa66f571154-data" (OuterVolumeSpecName: "data") pod "898072c6-66e4-4d46-9db9-5fa66f571154" (UID: "898072c6-66e4-4d46-9db9-5fa66f571154"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:05.572109 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.572087    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898072c6-66e4-4d46-9db9-5fa66f571154-kube-api-access-25kk5" (OuterVolumeSpecName: "kube-api-access-25kk5") pod "898072c6-66e4-4d46-9db9-5fa66f571154" (UID: "898072c6-66e4-4d46-9db9-5fa66f571154"). InnerVolumeSpecName "kube-api-access-25kk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:05.671017 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.670972    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kk5\" (UniqueName: \"kubernetes.io/projected/898072c6-66e4-4d46-9db9-5fa66f571154-kube-api-access-25kk5\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.671017 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.671011    2562 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/898072c6-66e4-4d46-9db9-5fa66f571154-data\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.878580 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.878497    2562 generic.go:356] "Generic (PLEG): container finished" podID="898072c6-66e4-4d46-9db9-5fa66f571154" containerID="f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f" exitCode=0
Sep 16 18:54:05.878580 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.878539    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"
Sep 16 18:54:05.878782 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.878586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6" event={"ID":"898072c6-66e4-4d46-9db9-5fa66f571154","Type":"ContainerDied","Data":"f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f"}
Sep 16 18:54:05.878782 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.878649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6" event={"ID":"898072c6-66e4-4d46-9db9-5fa66f571154","Type":"ContainerDied","Data":"888f435c5ffc0d99a389a46d35b5a1b15652f918c1b59c49cdfe9e3f370ab423"}
Sep 16 18:54:05.878782 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.878668    2562 scope.go:117] "RemoveContainer" containerID="f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f"
Sep 16 18:54:05.888929 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.888910    2562 scope.go:117] "RemoveContainer" containerID="f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f"
Sep 16 18:54:05.889221 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:54:05.889205    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f\": container with ID starting with f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f not found: ID does not exist" containerID="f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f"
Sep 16 18:54:05.889274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.889229    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f"} err="failed to get container status \"f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f\": rpc error: code = NotFound desc = could not find container \"f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f\": container with ID starting with f678451cb214042950836f21dc24f54e25f2a3da9816b508e25051dac0b9d41f not found: ID does not exist"
Sep 16 18:54:05.899512 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.899492    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"]
Sep 16 18:54:05.906120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:05.906091    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qt8p6"]
Sep 16 18:54:06.122130 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:06.122085    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898072c6-66e4-4d46-9db9-5fa66f571154" path="/var/lib/kubelet/pods/898072c6-66e4-4d46-9db9-5fa66f571154/volumes"
Sep 16 18:54:07.401511 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.401476    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-nnjzs"]
Sep 16 18:54:07.401954 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.401776    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898072c6-66e4-4d46-9db9-5fa66f571154" containerName="seaweedfs-tls-custom"
Sep 16 18:54:07.401954 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.401787    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="898072c6-66e4-4d46-9db9-5fa66f571154" containerName="seaweedfs-tls-custom"
Sep 16 18:54:07.401954 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.401853    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="898072c6-66e4-4d46-9db9-5fa66f571154" containerName="seaweedfs-tls-custom"
Sep 16 18:54:07.406359 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.406339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.408760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.408697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:54:07.408760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.408733    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:54:07.408760 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.408740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 18:54:07.409074 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.408778    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:54:07.410115 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.410085    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:54:07.411999 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.411955    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-nnjzs"]
Sep 16 18:54:07.486955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.486921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq97c\" (UniqueName: \"kubernetes.io/projected/d9e7d6e1-e949-4d54-999f-0326cba10469-kube-api-access-bq97c\") pod \"s3-tls-init-custom-nnjzs\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") " pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.486955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.486956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e7d6e1-e949-4d54-999f-0326cba10469-model-cache\") pod \"s3-tls-init-custom-nnjzs\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") " pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.587962 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.587925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq97c\" (UniqueName: \"kubernetes.io/projected/d9e7d6e1-e949-4d54-999f-0326cba10469-kube-api-access-bq97c\") pod \"s3-tls-init-custom-nnjzs\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") " pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.587962 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.587964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e7d6e1-e949-4d54-999f-0326cba10469-model-cache\") pod \"s3-tls-init-custom-nnjzs\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") " pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.588309 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.588293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e7d6e1-e949-4d54-999f-0326cba10469-model-cache\") pod \"s3-tls-init-custom-nnjzs\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") " pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.597445 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.597421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq97c\" (UniqueName: \"kubernetes.io/projected/d9e7d6e1-e949-4d54-999f-0326cba10469-kube-api-access-bq97c\") pod \"s3-tls-init-custom-nnjzs\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") " pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.717537 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.717457    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:07.986868 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:07.986794    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-nnjzs"]
Sep 16 18:54:07.990290 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:54:07.990257    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e7d6e1_e949_4d54_999f_0326cba10469.slice/crio-1c234ca24e223cf158e8ff0bdc357dc35875f9c9f9ab3ab5fb3968057bfd8eda WatchSource:0}: Error finding container 1c234ca24e223cf158e8ff0bdc357dc35875f9c9f9ab3ab5fb3968057bfd8eda: Status 404 returned error can't find the container with id 1c234ca24e223cf158e8ff0bdc357dc35875f9c9f9ab3ab5fb3968057bfd8eda
Sep 16 18:54:08.895466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:08.895376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nnjzs" event={"ID":"d9e7d6e1-e949-4d54-999f-0326cba10469","Type":"ContainerStarted","Data":"0daa89461fbfb03cfd96c5c983ddd66930d8d535f6da35cfe12e9542fb8b3b7a"}
Sep 16 18:54:08.895466 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:08.895426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nnjzs" event={"ID":"d9e7d6e1-e949-4d54-999f-0326cba10469","Type":"ContainerStarted","Data":"1c234ca24e223cf158e8ff0bdc357dc35875f9c9f9ab3ab5fb3968057bfd8eda"}
Sep 16 18:54:09.901086 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:09.901055    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerID="0daa89461fbfb03cfd96c5c983ddd66930d8d535f6da35cfe12e9542fb8b3b7a" exitCode=0
Sep 16 18:54:09.901470 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:09.901138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nnjzs" event={"ID":"d9e7d6e1-e949-4d54-999f-0326cba10469","Type":"ContainerDied","Data":"0daa89461fbfb03cfd96c5c983ddd66930d8d535f6da35cfe12e9542fb8b3b7a"}
Sep 16 18:54:10.906670 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:10.906633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nnjzs" event={"ID":"d9e7d6e1-e949-4d54-999f-0326cba10469","Type":"ContainerStarted","Data":"c798968fe19ed1de74651ff6ef129c7f180babd37e297c0827274fb69c228e65"}
Sep 16 18:54:10.928411 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:10.928364    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-nnjzs" podStartSLOduration=2.261258802 podStartE2EDuration="3.928349267s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.667090465s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:10.926084104 +0000 UTC m=+653.342786862" watchObservedRunningTime="2026-09-16 18:54:10.928349267 +0000 UTC m=+653.345052023"
Sep 16 18:54:16.933759 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:16.933721    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerID="c798968fe19ed1de74651ff6ef129c7f180babd37e297c0827274fb69c228e65" exitCode=0
Sep 16 18:54:16.934111 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:16.933777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nnjzs" event={"ID":"d9e7d6e1-e949-4d54-999f-0326cba10469","Type":"ContainerDied","Data":"c798968fe19ed1de74651ff6ef129c7f180babd37e297c0827274fb69c228e65"}
Sep 16 18:54:18.085416 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.085391    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:18.188419 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.188337    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e7d6e1-e949-4d54-999f-0326cba10469-model-cache\") pod \"d9e7d6e1-e949-4d54-999f-0326cba10469\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") "
Sep 16 18:54:18.188558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.188449    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq97c\" (UniqueName: \"kubernetes.io/projected/d9e7d6e1-e949-4d54-999f-0326cba10469-kube-api-access-bq97c\") pod \"d9e7d6e1-e949-4d54-999f-0326cba10469\" (UID: \"d9e7d6e1-e949-4d54-999f-0326cba10469\") "
Sep 16 18:54:18.190647 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.190611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e7d6e1-e949-4d54-999f-0326cba10469-kube-api-access-bq97c" (OuterVolumeSpecName: "kube-api-access-bq97c") pod "d9e7d6e1-e949-4d54-999f-0326cba10469" (UID: "d9e7d6e1-e949-4d54-999f-0326cba10469"). InnerVolumeSpecName "kube-api-access-bq97c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:18.211576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.211546    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e7d6e1-e949-4d54-999f-0326cba10469-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9e7d6e1-e949-4d54-999f-0326cba10469" (UID: "d9e7d6e1-e949-4d54-999f-0326cba10469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:18.289464 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.289422    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq97c\" (UniqueName: \"kubernetes.io/projected/d9e7d6e1-e949-4d54-999f-0326cba10469-kube-api-access-bq97c\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:54:18.289464 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.289460    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e7d6e1-e949-4d54-999f-0326cba10469-model-cache\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:54:18.943331 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.943280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-nnjzs" event={"ID":"d9e7d6e1-e949-4d54-999f-0326cba10469","Type":"ContainerDied","Data":"1c234ca24e223cf158e8ff0bdc357dc35875f9c9f9ab3ab5fb3968057bfd8eda"}
Sep 16 18:54:18.943506 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.943352    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c234ca24e223cf158e8ff0bdc357dc35875f9c9f9ab3ab5fb3968057bfd8eda"
Sep 16 18:54:18.943506 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:18.943291    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-nnjzs"
Sep 16 18:54:19.569675 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:19.569624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:21.057842 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.057805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:21.758594 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.758562    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-fl5tx"]
Sep 16 18:54:21.758904 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.758891    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerName="download-hf-model"
Sep 16 18:54:21.758947 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.758906    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerName="download-hf-model"
Sep 16 18:54:21.758947 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.758914    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerName="s3-tls-init-custom"
Sep 16 18:54:21.758947 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.758920    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerName="s3-tls-init-custom"
Sep 16 18:54:21.759044 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.758977    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e7d6e1-e949-4d54-999f-0326cba10469" containerName="s3-tls-init-custom"
Sep 16 18:54:21.762937 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.762922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:21.765274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.765249    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:54:21.765426 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.765275    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:54:21.765426 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.765298    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:54:21.765426 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.765360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 18:54:21.766136 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.766117    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:54:21.769304 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.769281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-fl5tx"]
Sep 16 18:54:21.823953 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.823922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c7cc52-6837-4c21-84e4-0cc10af90bae-model-cache\") pod \"s3-tls-init-serving-fl5tx\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") " pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:21.824127 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.823974    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tglqw\" (UniqueName: \"kubernetes.io/projected/07c7cc52-6837-4c21-84e4-0cc10af90bae-kube-api-access-tglqw\") pod \"s3-tls-init-serving-fl5tx\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") " pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:21.924890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.924857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tglqw\" (UniqueName: \"kubernetes.io/projected/07c7cc52-6837-4c21-84e4-0cc10af90bae-kube-api-access-tglqw\") pod \"s3-tls-init-serving-fl5tx\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") " pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:21.925032 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.924952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c7cc52-6837-4c21-84e4-0cc10af90bae-model-cache\") pod \"s3-tls-init-serving-fl5tx\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") " pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:21.925276 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.925260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c7cc52-6837-4c21-84e4-0cc10af90bae-model-cache\") pod \"s3-tls-init-serving-fl5tx\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") " pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:21.933418 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:21.933387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tglqw\" (UniqueName: \"kubernetes.io/projected/07c7cc52-6837-4c21-84e4-0cc10af90bae-kube-api-access-tglqw\") pod \"s3-tls-init-serving-fl5tx\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") " pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:22.073597 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:22.073551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:22.343001 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:22.342937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-fl5tx"]
Sep 16 18:54:22.345760 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:54:22.345731    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07c7cc52_6837_4c21_84e4_0cc10af90bae.slice/crio-cdbd67a08575666c2842eb09a9b179c65685850d5aa8abed0199b6f0be93e387 WatchSource:0}: Error finding container cdbd67a08575666c2842eb09a9b179c65685850d5aa8abed0199b6f0be93e387: Status 404 returned error can't find the container with id cdbd67a08575666c2842eb09a9b179c65685850d5aa8abed0199b6f0be93e387
Sep 16 18:54:22.961425 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:22.961392    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fl5tx" event={"ID":"07c7cc52-6837-4c21-84e4-0cc10af90bae","Type":"ContainerStarted","Data":"fd2487fb57532b73b0b82211acb159019b15eee55bb2343f7a7f9ef38b1a93a4"}
Sep 16 18:54:22.961601 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:22.961436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fl5tx" event={"ID":"07c7cc52-6837-4c21-84e4-0cc10af90bae","Type":"ContainerStarted","Data":"cdbd67a08575666c2842eb09a9b179c65685850d5aa8abed0199b6f0be93e387"}
Sep 16 18:54:24.970996 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:24.970964    2562 generic.go:356] "Generic (PLEG): container finished" podID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerID="fd2487fb57532b73b0b82211acb159019b15eee55bb2343f7a7f9ef38b1a93a4" exitCode=0
Sep 16 18:54:24.971400 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:24.971008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fl5tx" event={"ID":"07c7cc52-6837-4c21-84e4-0cc10af90bae","Type":"ContainerDied","Data":"fd2487fb57532b73b0b82211acb159019b15eee55bb2343f7a7f9ef38b1a93a4"}
Sep 16 18:54:25.977118 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:25.977083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fl5tx" event={"ID":"07c7cc52-6837-4c21-84e4-0cc10af90bae","Type":"ContainerStarted","Data":"aa22b2767eeb84988d310efa307307ca2b8cd86bbdb63a30c67c2706ca4ce307"}
Sep 16 18:54:25.999431 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:25.999384    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-fl5tx" podStartSLOduration=2.997534237 podStartE2EDuration="4.999369752s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.001835515s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:25.998270747 +0000 UTC m=+668.414973497" watchObservedRunningTime="2026-09-16 18:54:25.999369752 +0000 UTC m=+668.416072527"
Sep 16 18:54:33.012219 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:33.012175    2562 generic.go:356] "Generic (PLEG): container finished" podID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerID="aa22b2767eeb84988d310efa307307ca2b8cd86bbdb63a30c67c2706ca4ce307" exitCode=0
Sep 16 18:54:33.012650 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:33.012247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fl5tx" event={"ID":"07c7cc52-6837-4c21-84e4-0cc10af90bae","Type":"ContainerDied","Data":"aa22b2767eeb84988d310efa307307ca2b8cd86bbdb63a30c67c2706ca4ce307"}
Sep 16 18:54:34.166448 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.166418    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:34.338855 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.338820    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tglqw\" (UniqueName: \"kubernetes.io/projected/07c7cc52-6837-4c21-84e4-0cc10af90bae-kube-api-access-tglqw\") pod \"07c7cc52-6837-4c21-84e4-0cc10af90bae\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") "
Sep 16 18:54:34.339014 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.338921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c7cc52-6837-4c21-84e4-0cc10af90bae-model-cache\") pod \"07c7cc52-6837-4c21-84e4-0cc10af90bae\" (UID: \"07c7cc52-6837-4c21-84e4-0cc10af90bae\") "
Sep 16 18:54:34.341040 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.341004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c7cc52-6837-4c21-84e4-0cc10af90bae-kube-api-access-tglqw" (OuterVolumeSpecName: "kube-api-access-tglqw") pod "07c7cc52-6837-4c21-84e4-0cc10af90bae" (UID: "07c7cc52-6837-4c21-84e4-0cc10af90bae"). InnerVolumeSpecName "kube-api-access-tglqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:34.358244 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.358221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c7cc52-6837-4c21-84e4-0cc10af90bae-model-cache" (OuterVolumeSpecName: "model-cache") pod "07c7cc52-6837-4c21-84e4-0cc10af90bae" (UID: "07c7cc52-6837-4c21-84e4-0cc10af90bae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:34.439790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.439755    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tglqw\" (UniqueName: \"kubernetes.io/projected/07c7cc52-6837-4c21-84e4-0cc10af90bae-kube-api-access-tglqw\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:54:34.439790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:34.439786    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c7cc52-6837-4c21-84e4-0cc10af90bae-model-cache\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:54:35.026200 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:35.026166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fl5tx" event={"ID":"07c7cc52-6837-4c21-84e4-0cc10af90bae","Type":"ContainerDied","Data":"cdbd67a08575666c2842eb09a9b179c65685850d5aa8abed0199b6f0be93e387"}
Sep 16 18:54:35.026200 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:35.026188    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fl5tx"
Sep 16 18:54:35.026200 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:35.026204    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdbd67a08575666c2842eb09a9b179c65685850d5aa8abed0199b6f0be93e387"
Sep 16 18:54:35.397070 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:35.397038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:35.497268 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:35.497238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:52.393442 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:52.393408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:54.027962 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.027927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:54.152914 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.152880    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"]
Sep 16 18:54:54.153186 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.153174    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerName="s3-tls-init-serving"
Sep 16 18:54:54.153233 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.153188    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerName="s3-tls-init-serving"
Sep 16 18:54:54.153233 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.153198    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerName="download-hf-model"
Sep 16 18:54:54.153233 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.153204    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerName="download-hf-model"
Sep 16 18:54:54.153347 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.153261    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c7cc52-6837-4c21-84e4-0cc10af90bae" containerName="s3-tls-init-serving"
Sep 16 18:54:54.155543 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.155528    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.157908 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.157869    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 16 18:54:54.157908 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.157899    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:54:54.158127 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.157918    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 16 18:54:54.158127 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.157959    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 18:54:54.158280 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.158264    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:54:54.158381 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.158287    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:54:54.158767 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.158753    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:54:54.165498 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.165481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"]
Sep 16 18:54:54.198141 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.198115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.198257 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.198151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.198257 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.198182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.198391 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.198255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.198391 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.198347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrxh\" (UniqueName: \"kubernetes.io/projected/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-api-access-qhrxh\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.231581 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.231562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:54.298842 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.298817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.298941 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.298856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.298941 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.298877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.298941 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.298922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.299111 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.298969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrxh\" (UniqueName: \"kubernetes.io/projected/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-api-access-qhrxh\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.299111 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:54:54.298991    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 16 18:54:54.299111 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:54:54.299063    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls podName:7debcee8-f596-4de3-b5d6-9b3a161bfab4 nodeName:}" failed. No retries permitted until 2026-09-16 18:54:54.799042014 +0000 UTC m=+697.215744765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls") pod "isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" (UID: "7debcee8-f596-4de3-b5d6-9b3a161bfab4") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 16 18:54:54.299281 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.299245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.299589 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.299572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.299644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.299584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.307229 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.307211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrxh\" (UniqueName: \"kubernetes.io/projected/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-api-access-qhrxh\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.316044 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.316023    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"]
Sep 16 18:54:54.318453 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.318437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.320976 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.320950    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-7e78f2-dockercfg-bmh98\""
Sep 16 18:54:54.321081 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.320952    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7e78f2-kube-rbac-proxy-sar-config\""
Sep 16 18:54:54.321081 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.320997    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7e78f2-predictor-serving-cert\""
Sep 16 18:54:54.321081 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.321001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-7e78f2\""
Sep 16 18:54:54.332351 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.332303    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"]
Sep 16 18:54:54.398736 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.398701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:54.399779 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.399760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.399829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.399788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.399829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.399806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-cabundle-cert\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.399933 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.399916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088657d0-3bac-4dae-9d5c-31b63aba8ace-kserve-provision-location\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.399986 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.399973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhhr\" (UniqueName: \"kubernetes.io/projected/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-api-access-nbhhr\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501080 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088657d0-3bac-4dae-9d5c-31b63aba8ace-kserve-provision-location\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501243 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhhr\" (UniqueName: \"kubernetes.io/projected/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-api-access-nbhhr\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501243 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501243 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501243 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-cabundle-cert\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501474 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:54:54.501290    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-serving-cert: secret "isvc-init-fail-7e78f2-predictor-serving-cert" not found
Sep 16 18:54:54.501474 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:54:54.501393    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls podName:088657d0-3bac-4dae-9d5c-31b63aba8ace nodeName:}" failed. No retries permitted until 2026-09-16 18:54:55.001373553 +0000 UTC m=+697.418076291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls") pod "isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" (UID: "088657d0-3bac-4dae-9d5c-31b63aba8ace") : secret "isvc-init-fail-7e78f2-predictor-serving-cert" not found
Sep 16 18:54:54.501550 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088657d0-3bac-4dae-9d5c-31b63aba8ace-kserve-provision-location\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501932 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-cabundle-cert\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.501972 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.501911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.509761 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.509742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhhr\" (UniqueName: \"kubernetes.io/projected/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-api-access-nbhhr\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:54.576987 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.576901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:54.804742 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.804704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:54.807235 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:54.807215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:55.005964 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:55.005856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:55.008459 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:55.008437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls\") pod \"isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") " pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:55.067079 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:55.067041    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:54:55.228509 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:55.228454    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:54:55.341975 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:55.341951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"]
Sep 16 18:54:55.344495 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:54:55.344457    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7debcee8_f596_4de3_b5d6_9b3a161bfab4.slice/crio-c340e79814b9bf11370cd8ce367911d2264aca57d856e28061a379742b983911 WatchSource:0}: Error finding container c340e79814b9bf11370cd8ce367911d2264aca57d856e28061a379742b983911: Status 404 returned error can't find the container with id c340e79814b9bf11370cd8ce367911d2264aca57d856e28061a379742b983911
Sep 16 18:54:55.499075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:55.498919    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"]
Sep 16 18:54:55.501771 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:54:55.501742    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088657d0_3bac_4dae_9d5c_31b63aba8ace.slice/crio-45a1f6eed380cc3121d99056159a9fb61ee21c1e89bb2608a99abaef05ee969f WatchSource:0}: Error finding container 45a1f6eed380cc3121d99056159a9fb61ee21c1e89bb2608a99abaef05ee969f: Status 404 returned error can't find the container with id 45a1f6eed380cc3121d99056159a9fb61ee21c1e89bb2608a99abaef05ee969f
Sep 16 18:54:56.116398 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:56.116362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" event={"ID":"088657d0-3bac-4dae-9d5c-31b63aba8ace","Type":"ContainerStarted","Data":"b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b"}
Sep 16 18:54:56.116920 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:56.116494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" event={"ID":"088657d0-3bac-4dae-9d5c-31b63aba8ace","Type":"ContainerStarted","Data":"45a1f6eed380cc3121d99056159a9fb61ee21c1e89bb2608a99abaef05ee969f"}
Sep 16 18:54:56.121651 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:56.121624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerStarted","Data":"313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22"}
Sep 16 18:54:56.121789 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:54:56.121679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerStarted","Data":"c340e79814b9bf11370cd8ce367911d2264aca57d856e28061a379742b983911"}
Sep 16 18:55:00.135693 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:00.135668    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc_088657d0-3bac-4dae-9d5c-31b63aba8ace/storage-initializer/0.log"
Sep 16 18:55:00.136075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:00.135706    2562 generic.go:356] "Generic (PLEG): container finished" podID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerID="b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b" exitCode=1
Sep 16 18:55:00.136075 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:00.135775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" event={"ID":"088657d0-3bac-4dae-9d5c-31b63aba8ace","Type":"ContainerDied","Data":"b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b"}
Sep 16 18:55:00.137588 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:00.137502    2562 generic.go:356] "Generic (PLEG): container finished" podID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerID="313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22" exitCode=0
Sep 16 18:55:00.137588 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:00.137534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerDied","Data":"313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22"}
Sep 16 18:55:01.150097 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:01.150061    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc_088657d0-3bac-4dae-9d5c-31b63aba8ace/storage-initializer/0.log"
Sep 16 18:55:01.150598 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:01.150167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" event={"ID":"088657d0-3bac-4dae-9d5c-31b63aba8ace","Type":"ContainerStarted","Data":"f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0"}
Sep 16 18:55:04.052676 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:04.052636    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"]
Sep 16 18:55:04.053094 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:04.052966    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer" containerID="cri-o://f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0" gracePeriod=30
Sep 16 18:55:04.075148 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:04.075106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:05.140156 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.140131    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc_088657d0-3bac-4dae-9d5c-31b63aba8ace/storage-initializer/1.log"
Sep 16 18:55:05.140652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.140635    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc_088657d0-3bac-4dae-9d5c-31b63aba8ace/storage-initializer/0.log"
Sep 16 18:55:05.141776 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.141756    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:55:05.175843 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.175815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc_088657d0-3bac-4dae-9d5c-31b63aba8ace/storage-initializer/1.log"
Sep 16 18:55:05.176261 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.176242    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc_088657d0-3bac-4dae-9d5c-31b63aba8ace/storage-initializer/0.log"
Sep 16 18:55:05.176393 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.176284    2562 generic.go:356] "Generic (PLEG): container finished" podID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerID="f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0" exitCode=1
Sep 16 18:55:05.176393 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.176388    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"
Sep 16 18:55:05.176494 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.176407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" event={"ID":"088657d0-3bac-4dae-9d5c-31b63aba8ace","Type":"ContainerDied","Data":"f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0"}
Sep 16 18:55:05.176494 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.176477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc" event={"ID":"088657d0-3bac-4dae-9d5c-31b63aba8ace","Type":"ContainerDied","Data":"45a1f6eed380cc3121d99056159a9fb61ee21c1e89bb2608a99abaef05ee969f"}
Sep 16 18:55:05.176607 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.176517    2562 scope.go:117] "RemoveContainer" containerID="f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0"
Sep 16 18:55:05.186567 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.186549    2562 scope.go:117] "RemoveContainer" containerID="b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b"
Sep 16 18:55:05.197467 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.197424    2562 scope.go:117] "RemoveContainer" containerID="f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0"
Sep 16 18:55:05.197802 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:55:05.197778    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0\": container with ID starting with f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0 not found: ID does not exist" containerID="f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0"
Sep 16 18:55:05.197888 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.197813    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0"} err="failed to get container status \"f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0\": rpc error: code = NotFound desc = could not find container \"f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0\": container with ID starting with f18ac16b8fb132789c3db9215e26d8142e76c87e26f27bf3cd508c0f5ddc92e0 not found: ID does not exist"
Sep 16 18:55:05.197888 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.197832    2562 scope.go:117] "RemoveContainer" containerID="b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b"
Sep 16 18:55:05.198161 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:55:05.198140    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b\": container with ID starting with b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b not found: ID does not exist" containerID="b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b"
Sep 16 18:55:05.198229 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.198170    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b"} err="failed to get container status \"b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b\": rpc error: code = NotFound desc = could not find container \"b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b\": container with ID starting with b01bf106b18c3267fc9cb275aca81aef09258bac4fe74143052388ae5e6aa07b not found: ID does not exist"
Sep 16 18:55:05.203448 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203424    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088657d0-3bac-4dae-9d5c-31b63aba8ace-kserve-provision-location\") pod \"088657d0-3bac-4dae-9d5c-31b63aba8ace\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") "
Sep 16 18:55:05.203541 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls\") pod \"088657d0-3bac-4dae-9d5c-31b63aba8ace\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") "
Sep 16 18:55:05.203596 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-rbac-proxy-sar-config\") pod \"088657d0-3bac-4dae-9d5c-31b63aba8ace\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") "
Sep 16 18:55:05.203664 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-cabundle-cert\") pod \"088657d0-3bac-4dae-9d5c-31b63aba8ace\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") "
Sep 16 18:55:05.203664 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203628    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbhhr\" (UniqueName: \"kubernetes.io/projected/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-api-access-nbhhr\") pod \"088657d0-3bac-4dae-9d5c-31b63aba8ace\" (UID: \"088657d0-3bac-4dae-9d5c-31b63aba8ace\") "
Sep 16 18:55:05.203759 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088657d0-3bac-4dae-9d5c-31b63aba8ace-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "088657d0-3bac-4dae-9d5c-31b63aba8ace" (UID: "088657d0-3bac-4dae-9d5c-31b63aba8ace"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:05.203883 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203862    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/088657d0-3bac-4dae-9d5c-31b63aba8ace-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:55:05.203980 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.203930    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "088657d0-3bac-4dae-9d5c-31b63aba8ace" (UID: "088657d0-3bac-4dae-9d5c-31b63aba8ace"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:05.204066 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.204037    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "088657d0-3bac-4dae-9d5c-31b63aba8ace" (UID: "088657d0-3bac-4dae-9d5c-31b63aba8ace"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:05.206264 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.206238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "088657d0-3bac-4dae-9d5c-31b63aba8ace" (UID: "088657d0-3bac-4dae-9d5c-31b63aba8ace"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:05.206525 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.206504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-api-access-nbhhr" (OuterVolumeSpecName: "kube-api-access-nbhhr") pod "088657d0-3bac-4dae-9d5c-31b63aba8ace" (UID: "088657d0-3bac-4dae-9d5c-31b63aba8ace"). InnerVolumeSpecName "kube-api-access-nbhhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:05.304493 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.304459    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:55:05.304493 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.304493    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/088657d0-3bac-4dae-9d5c-31b63aba8ace-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:55:05.304679 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.304507    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbhhr\" (UniqueName: \"kubernetes.io/projected/088657d0-3bac-4dae-9d5c-31b63aba8ace-kube-api-access-nbhhr\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:55:05.304679 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.304518    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/088657d0-3bac-4dae-9d5c-31b63aba8ace-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:55:05.514467 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.514434    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"]
Sep 16 18:55:05.520644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:05.520621    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7e78f2-predictor-67df985b46-qfgdc"]
Sep 16 18:55:06.122912 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:06.122877    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" path="/var/lib/kubelet/pods/088657d0-3bac-4dae-9d5c-31b63aba8ace/volumes"
Sep 16 18:55:09.186955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.186915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:09.236889 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.236854    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"]
Sep 16 18:55:09.237279 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.237261    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer"
Sep 16 18:55:09.237382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.237281    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer"
Sep 16 18:55:09.237382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.237294    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer"
Sep 16 18:55:09.237382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.237299    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer"
Sep 16 18:55:09.237382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.237381    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer"
Sep 16 18:55:09.237565 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.237391    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="088657d0-3bac-4dae-9d5c-31b63aba8ace" containerName="storage-initializer"
Sep 16 18:55:09.248599 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.248563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.250895 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.250864    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 16 18:55:09.251019 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.250978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 18:55:09.251093 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.251062    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"]
Sep 16 18:55:09.274915 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.274887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:09.342071 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.342035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.342254 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.342085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.342254 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.342181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7s2b\" (UniqueName: \"kubernetes.io/projected/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-api-access-w7s2b\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.342374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.342282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.342374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.342343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.443592 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.443500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.443592 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.443554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.443592 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.443592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7s2b\" (UniqueName: \"kubernetes.io/projected/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-api-access-w7s2b\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.443863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.443667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.443863 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.443690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.443995 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.443972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.444453 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.444432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.444620 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.444595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.446584 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.446557    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.452716 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.452682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7s2b\" (UniqueName: \"kubernetes.io/projected/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-api-access-w7s2b\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-c6hpz\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:09.562877 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:09.562829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:13.336983 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:13.336953    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"]
Sep 16 18:55:13.338395 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:55:13.338365    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaed2ded_5a36_47f3_8654_73ef5b0dfb6e.slice/crio-9d095a4df6a05c8c203796ba2c8d868dfb25313d274e622296f989f24c75cf9d WatchSource:0}: Error finding container 9d095a4df6a05c8c203796ba2c8d868dfb25313d274e622296f989f24c75cf9d: Status 404 returned error can't find the container with id 9d095a4df6a05c8c203796ba2c8d868dfb25313d274e622296f989f24c75cf9d
Sep 16 18:55:14.225995 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:14.225955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerStarted","Data":"a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349"}
Sep 16 18:55:14.228309 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:14.228274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerStarted","Data":"2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b"}
Sep 16 18:55:14.228455 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:14.228418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerStarted","Data":"9d095a4df6a05c8c203796ba2c8d868dfb25313d274e622296f989f24c75cf9d"}
Sep 16 18:55:16.240061 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:16.240024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerStarted","Data":"ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc"}
Sep 16 18:55:18.250450 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:18.250415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerStarted","Data":"100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7"}
Sep 16 18:55:18.250890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:18.250609    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:55:18.251940 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:18.251916    2562 generic.go:356] "Generic (PLEG): container finished" podID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerID="2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b" exitCode=0
Sep 16 18:55:18.252066 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:18.251983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerDied","Data":"2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b"}
Sep 16 18:55:18.294738 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:18.294693    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podStartSLOduration=2.738664979 podStartE2EDuration="24.294676356s" totalImagesPullingTime="17.811953342s" totalInitContainerRuntime="3.744058035s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:18.294368268 +0000 UTC m=+720.711071029" watchObservedRunningTime="2026-09-16 18:55:18.294676356 +0000 UTC m=+720.711379115"
Sep 16 18:55:19.263168 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:19.263134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:55:19.263664 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:19.263305    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:55:19.265569 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:19.265019    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:19.266128 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:19.266091    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:20.271132 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:20.270761    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:20.271615 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:20.271361    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:20.275620 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:20.275565    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:55:21.276367 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:21.276038    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:21.276367 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:21.276279    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:31.276558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:31.276476    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:31.277174 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:31.277117    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:41.275997 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:41.275947    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:41.276555 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:41.276530    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:41.389119 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:41.389078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerStarted","Data":"4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a"}
Sep 16 18:55:41.389278 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:41.389139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerStarted","Data":"d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181"}
Sep 16 18:55:41.389365 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:41.389344    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:41.411085 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:41.411021    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podStartSLOduration=6.377159181 podStartE2EDuration="32.411003011s" totalImagesPullingTime="22.049104012s" totalInitContainerRuntime="3.984739818s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:41.410758865 +0000 UTC m=+743.827461625" watchObservedRunningTime="2026-09-16 18:55:41.411003011 +0000 UTC m=+743.827705773"
Sep 16 18:55:42.393547 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:42.393516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:42.394794 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:42.394766    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:55:43.398386 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:43.398347    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:55:48.402620 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:48.402579    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:55:48.403386 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:48.403354    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:55:51.276532 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:51.276488    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:55:51.277016 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:51.276992    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:55:56.370215 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:56.370181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:56.428624 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:56.428589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:58.403539 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:55:58.403499    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:01.276677 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:01.276635    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:56:01.277142 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:01.277112    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:08.403748 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:08.403704    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:11.276302 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:11.276255    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:56:11.276749 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:11.276712    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:18.403709 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:18.403668    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:21.276392 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:21.276305    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:56:21.276803 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:21.276783    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:28.403227 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:28.403180    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:31.276402 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:31.276351    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:56:31.276854 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:31.276834    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:38.404087 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:38.404040    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:41.276562 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:41.276474    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:56:41.277032 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:41.276791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:56:48.403963 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:48.403924    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:56:49.080962 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.080921    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"]
Sep 16 18:56:49.081293 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.081265    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" containerID="cri-o://a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349" gracePeriod=30
Sep 16 18:56:49.081406 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.081347    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" containerID="cri-o://100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7" gracePeriod=30
Sep 16 18:56:49.081480 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.081360    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" containerID="cri-o://ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc" gracePeriod=30
Sep 16 18:56:49.103551 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.103527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:49.155977 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.155951    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"]
Sep 16 18:56:49.158558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.158541    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.160584 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.160566    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 16 18:56:49.160744 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.160725    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 16 18:56:49.171684 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.171660    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"]
Sep 16 18:56:49.178433 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.178409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:49.232963 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.232935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zczdj\" (UniqueName: \"kubernetes.io/projected/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-api-access-zczdj\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.233082 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.232975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.233082 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.232999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e454ac-c9c0-4d63-b968-b5a04255da29-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.233082 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.233017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e454ac-c9c0-4d63-b968-b5a04255da29-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.233082 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.233072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.334576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.334496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e454ac-c9c0-4d63-b968-b5a04255da29-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.334576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.334529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e454ac-c9c0-4d63-b968-b5a04255da29-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.334576 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.334551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.334960 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.334627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zczdj\" (UniqueName: \"kubernetes.io/projected/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-api-access-zczdj\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.334960 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.334661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.334960 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.334928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e454ac-c9c0-4d63-b968-b5a04255da29-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.335417 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.335395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.335479 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.335425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.337048 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.337024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e454ac-c9c0-4d63-b968-b5a04255da29-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.344304 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.344284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zczdj\" (UniqueName: \"kubernetes.io/projected/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-api-access-zczdj\") pod \"isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.468658 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.468621    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:49.693054 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.692968    2562 generic.go:356] "Generic (PLEG): container finished" podID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerID="ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc" exitCode=2
Sep 16 18:56:49.693054 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.693033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerDied","Data":"ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc"}
Sep 16 18:56:49.747844 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:49.747820    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"]
Sep 16 18:56:49.749716 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:56:49.749692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e454ac_c9c0_4d63_b968_b5a04255da29.slice/crio-eda34073d41e7982929f14891e32ac813afb125faa95d59937e2cee4d29a08f2 WatchSource:0}: Error finding container eda34073d41e7982929f14891e32ac813afb125faa95d59937e2cee4d29a08f2: Status 404 returned error can't find the container with id eda34073d41e7982929f14891e32ac813afb125faa95d59937e2cee4d29a08f2
Sep 16 18:56:50.270672 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:50.270624    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:56:50.700703 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:50.700661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerStarted","Data":"cc0bc0034648ef6f28f78d34078733473459660dd57e840d5f77eeaeec3e8236"}
Sep 16 18:56:50.701106 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:50.700757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerStarted","Data":"eda34073d41e7982929f14891e32ac813afb125faa95d59937e2cee4d29a08f2"}
Sep 16 18:56:51.276463 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:51.276407    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:56:51.276784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:51.276760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:53.719016 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:53.718988    2562 generic.go:356] "Generic (PLEG): container finished" podID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerID="cc0bc0034648ef6f28f78d34078733473459660dd57e840d5f77eeaeec3e8236" exitCode=0
Sep 16 18:56:53.719439 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:53.719068    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerDied","Data":"cc0bc0034648ef6f28f78d34078733473459660dd57e840d5f77eeaeec3e8236"}
Sep 16 18:56:54.726403 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.726373    2562 generic.go:356] "Generic (PLEG): container finished" podID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerID="a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349" exitCode=0
Sep 16 18:56:54.726810 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.726445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerDied","Data":"a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349"}
Sep 16 18:56:54.729023 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.729003    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerStarted","Data":"090f89c0c52a33d26d821aca5f571a147d2661d3707030514af205c615ec6ef6"}
Sep 16 18:56:54.729123 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.729054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerStarted","Data":"81efa70b18b66611ea57325babe2481bc8020115f954c5274578a4cdb85a2de6"}
Sep 16 18:56:54.729123 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.729080    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerStarted","Data":"1f08b32ad97cf3363e32e8a5f7820c31e47a2db22a0f9d946885fabc67f18224"}
Sep 16 18:56:54.729378 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.729351    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:54.729514 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.729499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:54.730917 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.730895    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:56:54.749660 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:54.749621    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podStartSLOduration=2.195269309 podStartE2EDuration="5.749608706s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554339397s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:54.747934533 +0000 UTC m=+817.164637297" watchObservedRunningTime="2026-09-16 18:56:54.749608706 +0000 UTC m=+817.166311464"
Sep 16 18:56:55.271420 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:55.271373    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:56:55.733539 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:55.733514    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:56:55.733993 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:55.733499    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:56:55.734699 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:55.734674    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:56.738526 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:56.738484    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:56:56.738887 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:56.738823    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:58.404094 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:56:58.404057    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:57:00.271185 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:00.271148    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:57:00.271599 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:00.271259    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:57:01.276391 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:01.276312    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:57:01.276820 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:01.276715    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:01.743180 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:01.743151    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:57:01.743747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:01.743717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:57:01.744064 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:01.744045    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:03.251955 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.251917    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"]
Sep 16 18:57:03.252314 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.252218    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container" containerID="cri-o://d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181" gracePeriod=30
Sep 16 18:57:03.252314 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.252290    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kube-rbac-proxy" containerID="cri-o://4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a" gracePeriod=30
Sep 16 18:57:03.282232 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.282207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:03.399156 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.399117    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 18:57:03.773911 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.773879    2562 generic.go:356] "Generic (PLEG): container finished" podID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerID="4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a" exitCode=2
Sep 16 18:57:03.774099 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:03.773959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerDied","Data":"4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a"}
Sep 16 18:57:05.270710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:05.270670    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:57:07.992009 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:07.991988    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:57:08.096982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.096900    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7s2b\" (UniqueName: \"kubernetes.io/projected/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-api-access-w7s2b\") pod \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") "
Sep 16 18:57:08.096982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.096944    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-proxy-tls\") pod \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") "
Sep 16 18:57:08.096982 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.096975    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-cabundle-cert\") pod \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") "
Sep 16 18:57:08.097255 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.097063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-rbac-proxy-sar-config\") pod \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") "
Sep 16 18:57:08.097255 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.097099    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kserve-provision-location\") pod \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\" (UID: \"caed2ded-5a36-47f3-8654-73ef5b0dfb6e\") "
Sep 16 18:57:08.097465 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.097440    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "caed2ded-5a36-47f3-8654-73ef5b0dfb6e" (UID: "caed2ded-5a36-47f3-8654-73ef5b0dfb6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:08.097642 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.097595    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "caed2ded-5a36-47f3-8654-73ef5b0dfb6e" (UID: "caed2ded-5a36-47f3-8654-73ef5b0dfb6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:08.097801 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.097778    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "caed2ded-5a36-47f3-8654-73ef5b0dfb6e" (UID: "caed2ded-5a36-47f3-8654-73ef5b0dfb6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:08.099264 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.099240    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-api-access-w7s2b" (OuterVolumeSpecName: "kube-api-access-w7s2b") pod "caed2ded-5a36-47f3-8654-73ef5b0dfb6e" (UID: "caed2ded-5a36-47f3-8654-73ef5b0dfb6e"). InnerVolumeSpecName "kube-api-access-w7s2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:08.099446 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.099386    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "caed2ded-5a36-47f3-8654-73ef5b0dfb6e" (UID: "caed2ded-5a36-47f3-8654-73ef5b0dfb6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:08.198691 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.198657    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:08.198691 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.198684    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:08.198691 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.198694    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7s2b\" (UniqueName: \"kubernetes.io/projected/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-kube-api-access-w7s2b\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:08.198900 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.198703    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:08.198900 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.198712    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caed2ded-5a36-47f3-8654-73ef5b0dfb6e-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:08.465883 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.465781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:08.505276 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505244    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"]
Sep 16 18:57:08.505642 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505628    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="storage-initializer"
Sep 16 18:57:08.505689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505645    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="storage-initializer"
Sep 16 18:57:08.505689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505663    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container"
Sep 16 18:57:08.505689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505669    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container"
Sep 16 18:57:08.505689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505678    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kube-rbac-proxy"
Sep 16 18:57:08.505689 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505684    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kube-rbac-proxy"
Sep 16 18:57:08.505837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505742    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kserve-container"
Sep 16 18:57:08.505837 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.505753    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerName="kube-rbac-proxy"
Sep 16 18:57:08.507979 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.507962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.510128 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.510109    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 16 18:57:08.510236 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.510114    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 18:57:08.523932 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.523907    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"]
Sep 16 18:57:08.575012 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.574954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:08.601878 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.601844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.602043 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.601891    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.602043 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.601913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.602146 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.602070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2163c7-d8d6-46a5-82bf-d1b4510b478f-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.602146 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.602129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqt8\" (UniqueName: \"kubernetes.io/projected/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-api-access-ptqt8\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.703452 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.703421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2163c7-d8d6-46a5-82bf-d1b4510b478f-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.703640 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.703465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqt8\" (UniqueName: \"kubernetes.io/projected/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-api-access-ptqt8\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.703640 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.703503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.703640 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.703541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.703640 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.703571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.704000 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.703963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.704357 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.704330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.704457 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.704383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.706181 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.706164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2163c7-d8d6-46a5-82bf-d1b4510b478f-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.712795 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.712776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqt8\" (UniqueName: \"kubernetes.io/projected/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-api-access-ptqt8\") pod \"isvc-predictive-xgboost-predictor-695f978978-7xp24\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.800073 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.800044    2562 generic.go:356] "Generic (PLEG): container finished" podID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" containerID="d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181" exitCode=0
Sep 16 18:57:08.800197 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.800105    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"
Sep 16 18:57:08.800197 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.800121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerDied","Data":"d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181"}
Sep 16 18:57:08.800197 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.800182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz" event={"ID":"caed2ded-5a36-47f3-8654-73ef5b0dfb6e","Type":"ContainerDied","Data":"9d095a4df6a05c8c203796ba2c8d868dfb25313d274e622296f989f24c75cf9d"}
Sep 16 18:57:08.800349 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.800215    2562 scope.go:117] "RemoveContainer" containerID="4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a"
Sep 16 18:57:08.814805 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.814786    2562 scope.go:117] "RemoveContainer" containerID="d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181"
Sep 16 18:57:08.818061 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.818026    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:08.823360 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.823337    2562 scope.go:117] "RemoveContainer" containerID="2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b"
Sep 16 18:57:08.823810 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.823791    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"]
Sep 16 18:57:08.827417 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.827395    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-c6hpz"]
Sep 16 18:57:08.845631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.845612    2562 scope.go:117] "RemoveContainer" containerID="4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a"
Sep 16 18:57:08.846042 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:08.846023    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a\": container with ID starting with 4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a not found: ID does not exist" containerID="4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a"
Sep 16 18:57:08.846106 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.846055    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a"} err="failed to get container status \"4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a\": rpc error: code = NotFound desc = could not find container \"4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a\": container with ID starting with 4385657234ac5ff8159885e3050f45586c90173bd350f64719fb826478c29d1a not found: ID does not exist"
Sep 16 18:57:08.846106 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.846075    2562 scope.go:117] "RemoveContainer" containerID="d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181"
Sep 16 18:57:08.846443 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:08.846412    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181\": container with ID starting with d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181 not found: ID does not exist" containerID="d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181"
Sep 16 18:57:08.846510 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.846447    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181"} err="failed to get container status \"d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181\": rpc error: code = NotFound desc = could not find container \"d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181\": container with ID starting with d0472bfd04e8728d14d117a1a58491bd46676d6f4d5efb53d81c24d79ee87181 not found: ID does not exist"
Sep 16 18:57:08.846510 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.846464    2562 scope.go:117] "RemoveContainer" containerID="2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b"
Sep 16 18:57:08.846800 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:08.846783    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b\": container with ID starting with 2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b not found: ID does not exist" containerID="2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b"
Sep 16 18:57:08.846846 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:08.846805    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b"} err="failed to get container status \"2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b\": rpc error: code = NotFound desc = could not find container \"2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b\": container with ID starting with 2775aa932dcdf1b962c8fb8159abfa87178bf70e2086a37f2d2690d4cdb9398b not found: ID does not exist"
Sep 16 18:57:09.093152 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:09.092810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"]
Sep 16 18:57:09.095454 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:57:09.095425    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2163c7_d8d6_46a5_82bf_d1b4510b478f.slice/crio-36274910d26da66bfb9b636ceb7f3db9874277fcb9fbe58df29699a28772c312 WatchSource:0}: Error finding container 36274910d26da66bfb9b636ceb7f3db9874277fcb9fbe58df29699a28772c312: Status 404 returned error can't find the container with id 36274910d26da66bfb9b636ceb7f3db9874277fcb9fbe58df29699a28772c312
Sep 16 18:57:09.806719 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:09.806679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerStarted","Data":"66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3"}
Sep 16 18:57:09.806906 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:09.806791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerStarted","Data":"36274910d26da66bfb9b636ceb7f3db9874277fcb9fbe58df29699a28772c312"}
Sep 16 18:57:10.122812 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:10.122730    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caed2ded-5a36-47f3-8654-73ef5b0dfb6e" path="/var/lib/kubelet/pods/caed2ded-5a36-47f3-8654-73ef5b0dfb6e/volumes"
Sep 16 18:57:10.271513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:10.271471    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:57:11.276160 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:11.276115    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:57:11.276612 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:11.276268    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:57:11.276612 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:11.276419    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:11.276612 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:11.276487    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:57:11.743667 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:11.743628    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:57:11.744488 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:11.744459    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:13.827691 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:13.827655    2562 generic.go:356] "Generic (PLEG): container finished" podID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerID="66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3" exitCode=0
Sep 16 18:57:13.828078 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:13.827727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerDied","Data":"66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3"}
Sep 16 18:57:14.836052 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:14.836017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerStarted","Data":"a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0"}
Sep 16 18:57:14.836497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:14.836079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerStarted","Data":"19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f"}
Sep 16 18:57:14.836497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:14.836355    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:14.836596 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:14.836503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:14.837695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:14.837672    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:14.857624 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:14.857547    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podStartSLOduration=2.866488151 podStartE2EDuration="6.857525267s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.991037116s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:14.853866493 +0000 UTC m=+837.270569252" watchObservedRunningTime="2026-09-16 18:57:14.857525267 +0000 UTC m=+837.274228028"
Sep 16 18:57:15.271563 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:15.271464    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:57:15.842281 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:15.842240    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:19.226111 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.226081    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:57:19.292294 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kserve-provision-location\") pod \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") "
Sep 16 18:57:19.292430 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292300    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls\") pod \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") "
Sep 16 18:57:19.292430 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhrxh\" (UniqueName: \"kubernetes.io/projected/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-api-access-qhrxh\") pod \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") "
Sep 16 18:57:19.292535 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292432    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-cabundle-cert\") pod \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") "
Sep 16 18:57:19.292535 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292468    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-rbac-proxy-sar-config\") pod \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\" (UID: \"7debcee8-f596-4de3-b5d6-9b3a161bfab4\") "
Sep 16 18:57:19.292640 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292590    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7debcee8-f596-4de3-b5d6-9b3a161bfab4" (UID: "7debcee8-f596-4de3-b5d6-9b3a161bfab4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:19.292814 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292786    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7debcee8-f596-4de3-b5d6-9b3a161bfab4" (UID: "7debcee8-f596-4de3-b5d6-9b3a161bfab4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:19.292814 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292806    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:19.292936 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.292835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7debcee8-f596-4de3-b5d6-9b3a161bfab4" (UID: "7debcee8-f596-4de3-b5d6-9b3a161bfab4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:19.294614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.294589    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-api-access-qhrxh" (OuterVolumeSpecName: "kube-api-access-qhrxh") pod "7debcee8-f596-4de3-b5d6-9b3a161bfab4" (UID: "7debcee8-f596-4de3-b5d6-9b3a161bfab4"). InnerVolumeSpecName "kube-api-access-qhrxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:19.294614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.294602    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7debcee8-f596-4de3-b5d6-9b3a161bfab4" (UID: "7debcee8-f596-4de3-b5d6-9b3a161bfab4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:19.393566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.393502    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrxh\" (UniqueName: \"kubernetes.io/projected/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-api-access-qhrxh\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:19.393566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.393526    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:19.393566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.393535    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7debcee8-f596-4de3-b5d6-9b3a161bfab4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:19.393566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.393545    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7debcee8-f596-4de3-b5d6-9b3a161bfab4-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:57:19.863598 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.863562    2562 generic.go:356] "Generic (PLEG): container finished" podID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerID="100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7" exitCode=0
Sep 16 18:57:19.863758 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.863622    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"
Sep 16 18:57:19.863758 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.863640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerDied","Data":"100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7"}
Sep 16 18:57:19.863758 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.863713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f" event={"ID":"7debcee8-f596-4de3-b5d6-9b3a161bfab4","Type":"ContainerDied","Data":"c340e79814b9bf11370cd8ce367911d2264aca57d856e28061a379742b983911"}
Sep 16 18:57:19.863758 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.863744    2562 scope.go:117] "RemoveContainer" containerID="100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7"
Sep 16 18:57:19.875248 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.875231    2562 scope.go:117] "RemoveContainer" containerID="ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc"
Sep 16 18:57:19.882779 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.882762    2562 scope.go:117] "RemoveContainer" containerID="a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349"
Sep 16 18:57:19.887687 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.887657    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"]
Sep 16 18:57:19.890813 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.890792    2562 scope.go:117] "RemoveContainer" containerID="313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22"
Sep 16 18:57:19.892534 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.892515    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7946b7fbfb-85j2f"]
Sep 16 18:57:19.899409 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.899393    2562 scope.go:117] "RemoveContainer" containerID="100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7"
Sep 16 18:57:19.899703 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:19.899677    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7\": container with ID starting with 100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7 not found: ID does not exist" containerID="100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7"
Sep 16 18:57:19.899794 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.899711    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7"} err="failed to get container status \"100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7\": rpc error: code = NotFound desc = could not find container \"100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7\": container with ID starting with 100ea1a629249dcf95b0638c329ff850bd90688b156cfb006aefd1f23af4d9d7 not found: ID does not exist"
Sep 16 18:57:19.899794 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.899730    2562 scope.go:117] "RemoveContainer" containerID="ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc"
Sep 16 18:57:19.900008 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:19.899991    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc\": container with ID starting with ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc not found: ID does not exist" containerID="ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc"
Sep 16 18:57:19.900059 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.900021    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc"} err="failed to get container status \"ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc\": rpc error: code = NotFound desc = could not find container \"ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc\": container with ID starting with ad13709492e7d2100cb917f435e5af73115899a3f0bfb250b7e4c64c399abdbc not found: ID does not exist"
Sep 16 18:57:19.900059 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.900033    2562 scope.go:117] "RemoveContainer" containerID="a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349"
Sep 16 18:57:19.900357 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:19.900312    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349\": container with ID starting with a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349 not found: ID does not exist" containerID="a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349"
Sep 16 18:57:19.900427 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.900367    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349"} err="failed to get container status \"a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349\": rpc error: code = NotFound desc = could not find container \"a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349\": container with ID starting with a1c91a74a176ca9184065d147112547c1c6f860292a1a60f016e9cc719681349 not found: ID does not exist"
Sep 16 18:57:19.900427 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.900407    2562 scope.go:117] "RemoveContainer" containerID="313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22"
Sep 16 18:57:19.900725 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:57:19.900707    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22\": container with ID starting with 313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22 not found: ID does not exist" containerID="313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22"
Sep 16 18:57:19.900816 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:19.900726    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22"} err="failed to get container status \"313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22\": rpc error: code = NotFound desc = could not find container \"313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22\": container with ID starting with 313e622ef5ef9f7faae85e33601d9f0ba70da0cd6ddaf80edca3ba884e66ae22 not found: ID does not exist"
Sep 16 18:57:20.122880 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:20.122806    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" path="/var/lib/kubelet/pods/7debcee8-f596-4de3-b5d6-9b3a161bfab4/volumes"
Sep 16 18:57:20.847274 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:20.847245    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:57:20.847800 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:20.847774    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:21.744234 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:21.744189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:57:21.744716 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:21.744696    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:30.848173 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:30.848134    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:31.744529 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:31.744493    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:57:31.745039 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:31.745014    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:40.848605 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:40.848559    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:41.743752 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:41.743715    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:57:41.744151 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:41.744129    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:50.848043 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:50.848006    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:57:51.743719 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:51.743680    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:57:51.744164 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:57:51.744140    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:00.848504 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:00.848465    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:58:01.744383 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:01.744338    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:58:01.744748 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:01.744720    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:10.847915 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:10.847829    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:58:11.744500 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:11.744471    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:11.744903 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:11.744878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:20.848364 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:20.848300    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:58:24.224817 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.224784    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"]
Sep 16 18:58:24.225282 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.225135    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" containerID="cri-o://1f08b32ad97cf3363e32e8a5f7820c31e47a2db22a0f9d946885fabc67f18224" gracePeriod=30
Sep 16 18:58:24.225282 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.225182    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" containerID="cri-o://090f89c0c52a33d26d821aca5f571a147d2661d3707030514af205c615ec6ef6" gracePeriod=30
Sep 16 18:58:24.225282 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.225216    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" containerID="cri-o://81efa70b18b66611ea57325babe2481bc8020115f954c5274578a4cdb85a2de6" gracePeriod=30
Sep 16 18:58:24.309981 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.309945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.363346 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363304    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"]
Sep 16 18:58:24.363691 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363679    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363693    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363705    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363710    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363718    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="storage-initializer"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363725    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="storage-initializer"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363734    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent"
Sep 16 18:58:24.363737 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363739    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent"
Sep 16 18:58:24.363974 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363801    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kserve-container"
Sep 16 18:58:24.363974 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363808    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="agent"
Sep 16 18:58:24.363974 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.363818    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7debcee8-f596-4de3-b5d6-9b3a161bfab4" containerName="kube-rbac-proxy"
Sep 16 18:58:24.366844 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.366825    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.368949 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.368929    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 16 18:58:24.369050 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.368965    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 16 18:58:24.378614 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.378588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"]
Sep 16 18:58:24.390655 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.390610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.455811 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.455781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f46fde-990b-4908-a9f9-162c9de371e8-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.455992 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.455826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.455992 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.455963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvhq\" (UniqueName: \"kubernetes.io/projected/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-api-access-jjvhq\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.506989 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.506909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.557301 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.557268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvhq\" (UniqueName: \"kubernetes.io/projected/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-api-access-jjvhq\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.557472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.557346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f46fde-990b-4908-a9f9-162c9de371e8-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.557472 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.557379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.558084 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.558054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.560040 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.560019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f46fde-990b-4908-a9f9-162c9de371e8-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.565295 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.565273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvhq\" (UniqueName: \"kubernetes.io/projected/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-api-access-jjvhq\") pod \"message-dumper-predictor-6575bcfdfd-nb6l2\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.677141 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.677107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:24.949370 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.949337    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"]
Sep 16 18:58:24.952473 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:58:24.952445    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f46fde_990b_4908_a9f9_162c9de371e8.slice/crio-e6e54ab69c00ee0bd1bdd16eeb209689c2d87e97c074c5fdaf61c7dc2fbdfe49 WatchSource:0}: Error finding container e6e54ab69c00ee0bd1bdd16eeb209689c2d87e97c074c5fdaf61c7dc2fbdfe49: Status 404 returned error can't find the container with id e6e54ab69c00ee0bd1bdd16eeb209689c2d87e97c074c5fdaf61c7dc2fbdfe49
Sep 16 18:58:24.954287 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:24.954270    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:58:25.166101 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:25.166069    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" event={"ID":"d9f46fde-990b-4908-a9f9-162c9de371e8","Type":"ContainerStarted","Data":"e6e54ab69c00ee0bd1bdd16eeb209689c2d87e97c074c5fdaf61c7dc2fbdfe49"}
Sep 16 18:58:25.168596 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:25.168571    2562 generic.go:356] "Generic (PLEG): container finished" podID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerID="81efa70b18b66611ea57325babe2481bc8020115f954c5274578a4cdb85a2de6" exitCode=2
Sep 16 18:58:25.168726 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:25.168604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerDied","Data":"81efa70b18b66611ea57325babe2481bc8020115f954c5274578a4cdb85a2de6"}
Sep 16 18:58:26.739623 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:26.739583    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:27.179245 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:27.179208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" event={"ID":"d9f46fde-990b-4908-a9f9-162c9de371e8","Type":"ContainerStarted","Data":"9d211823285b507aaa5b7f63ce2a05ae653c948377e44482963156648101c332"}
Sep 16 18:58:27.179429 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:27.179412    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:27.179497 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:27.179457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" event={"ID":"d9f46fde-990b-4908-a9f9-162c9de371e8","Type":"ContainerStarted","Data":"ef6f008894899fde2800df9a8532737f35d99970b5a0e86deadfc7d9c46f93ce"}
Sep 16 18:58:27.195829 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:27.195789    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" podStartSLOduration=1.9914676390000001 podStartE2EDuration="3.195776445s" totalImagesPullingTime="1.204308806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:27.194497349 +0000 UTC m=+909.611200133" watchObservedRunningTime="2026-09-16 18:58:27.195776445 +0000 UTC m=+909.612479277"
Sep 16 18:58:28.183487 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:28.183448    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:28.184961 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:28.184937    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:29.189972 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:29.189894    2562 generic.go:356] "Generic (PLEG): container finished" podID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerID="1f08b32ad97cf3363e32e8a5f7820c31e47a2db22a0f9d946885fabc67f18224" exitCode=0
Sep 16 18:58:29.190354 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:29.189967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerDied","Data":"1f08b32ad97cf3363e32e8a5f7820c31e47a2db22a0f9d946885fabc67f18224"}
Sep 16 18:58:30.848923 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:30.848893    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:58:31.739382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:31.739311    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:31.743661 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:31.743639    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:58:31.745920 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:31.745895    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:35.200041 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:35.200011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 18:58:36.739398 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:36.739354    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:36.739842 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:36.739556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:38.515542 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:38.515508    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"]
Sep 16 18:58:38.516042 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:38.515764    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" containerID="cri-o://19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f" gracePeriod=30
Sep 16 18:58:38.516042 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:38.515850    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kube-rbac-proxy" containerID="cri-o://a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0" gracePeriod=30
Sep 16 18:58:38.544624 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:38.544596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:39.235194 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:39.235158    2562 generic.go:356] "Generic (PLEG): container finished" podID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerID="a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0" exitCode=2
Sep 16 18:58:39.235402 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:39.235235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerDied","Data":"a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0"}
Sep 16 18:58:40.843459 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:40.843412    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 18:58:40.847768 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:40.847743    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:58:41.738705 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:41.738658    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:41.744122 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:41.744098    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:58:41.745594 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:41.745570    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:43.052558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.052532    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:58:43.217944 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.217864    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-cabundle-cert\") pod \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") "
Sep 16 18:58:43.217944 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.217933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptqt8\" (UniqueName: \"kubernetes.io/projected/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-api-access-ptqt8\") pod \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") "
Sep 16 18:58:43.218188 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.217957    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-rbac-proxy-sar-config\") pod \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") "
Sep 16 18:58:43.218188 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.217975    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2163c7-d8d6-46a5-82bf-d1b4510b478f-proxy-tls\") pod \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") "
Sep 16 18:58:43.218188 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.218017    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kserve-provision-location\") pod \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\" (UID: \"be2163c7-d8d6-46a5-82bf-d1b4510b478f\") "
Sep 16 18:58:43.218436 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.218408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be2163c7-d8d6-46a5-82bf-d1b4510b478f" (UID: "be2163c7-d8d6-46a5-82bf-d1b4510b478f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:43.218501 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.218430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be2163c7-d8d6-46a5-82bf-d1b4510b478f" (UID: "be2163c7-d8d6-46a5-82bf-d1b4510b478f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:43.218501 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.218409    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be2163c7-d8d6-46a5-82bf-d1b4510b478f" (UID: "be2163c7-d8d6-46a5-82bf-d1b4510b478f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:43.220294 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.220267    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2163c7-d8d6-46a5-82bf-d1b4510b478f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be2163c7-d8d6-46a5-82bf-d1b4510b478f" (UID: "be2163c7-d8d6-46a5-82bf-d1b4510b478f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:43.220435 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.220308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-api-access-ptqt8" (OuterVolumeSpecName: "kube-api-access-ptqt8") pod "be2163c7-d8d6-46a5-82bf-d1b4510b478f" (UID: "be2163c7-d8d6-46a5-82bf-d1b4510b478f"). InnerVolumeSpecName "kube-api-access-ptqt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:43.254536 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.254510    2562 generic.go:356] "Generic (PLEG): container finished" podID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerID="19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f" exitCode=0
Sep 16 18:58:43.254637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.254542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerDied","Data":"19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f"}
Sep 16 18:58:43.254637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.254561    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"
Sep 16 18:58:43.254637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.254583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24" event={"ID":"be2163c7-d8d6-46a5-82bf-d1b4510b478f","Type":"ContainerDied","Data":"36274910d26da66bfb9b636ceb7f3db9874277fcb9fbe58df29699a28772c312"}
Sep 16 18:58:43.254637 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.254618    2562 scope.go:117] "RemoveContainer" containerID="a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0"
Sep 16 18:58:43.264152 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.264135    2562 scope.go:117] "RemoveContainer" containerID="19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f"
Sep 16 18:58:43.272012 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.271993    2562 scope.go:117] "RemoveContainer" containerID="66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3"
Sep 16 18:58:43.278197 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.278176    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"]
Sep 16 18:58:43.279710 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.279642    2562 scope.go:117] "RemoveContainer" containerID="a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0"
Sep 16 18:58:43.279994 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:43.279939    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0\": container with ID starting with a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0 not found: ID does not exist" containerID="a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0"
Sep 16 18:58:43.280099 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.279997    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0"} err="failed to get container status \"a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0\": rpc error: code = NotFound desc = could not find container \"a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0\": container with ID starting with a6285884fb1e12090d103102d24487aeeb36eb32fa8257bdeaeb7d7b5ea112a0 not found: ID does not exist"
Sep 16 18:58:43.280099 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.280023    2562 scope.go:117] "RemoveContainer" containerID="19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f"
Sep 16 18:58:43.280470 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:43.280444    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f\": container with ID starting with 19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f not found: ID does not exist" containerID="19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f"
Sep 16 18:58:43.280554 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.280479    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f"} err="failed to get container status \"19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f\": rpc error: code = NotFound desc = could not find container \"19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f\": container with ID starting with 19b874528f57fb7df139913e2543bee1c179b51fb094344eae6df964ccf2078f not found: ID does not exist"
Sep 16 18:58:43.280554 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.280501    2562 scope.go:117] "RemoveContainer" containerID="66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3"
Sep 16 18:58:43.280842 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:43.280821    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3\": container with ID starting with 66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3 not found: ID does not exist" containerID="66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3"
Sep 16 18:58:43.280894 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.280847    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3"} err="failed to get container status \"66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3\": rpc error: code = NotFound desc = could not find container \"66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3\": container with ID starting with 66a0109f5d93c78488c822a179e1665a793bf4f06ebf82db40e8e94fd81993f3 not found: ID does not exist"
Sep 16 18:58:43.282380 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.282363    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7xp24"]
Sep 16 18:58:43.318748 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.318726    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:43.318748 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.318747    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:43.318890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.318758    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqt8\" (UniqueName: \"kubernetes.io/projected/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-api-access-ptqt8\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:43.318890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.318767    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be2163c7-d8d6-46a5-82bf-d1b4510b478f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:43.318890 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.318776    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be2163c7-d8d6-46a5-82bf-d1b4510b478f-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:43.691844 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.691814    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:43.720843 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.720812    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"]
Sep 16 18:58:43.721187 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721172    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="storage-initializer"
Sep 16 18:58:43.721240 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721188    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="storage-initializer"
Sep 16 18:58:43.721240 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721196    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kube-rbac-proxy"
Sep 16 18:58:43.721240 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721202    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kube-rbac-proxy"
Sep 16 18:58:43.721240 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721216    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container"
Sep 16 18:58:43.721240 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721222    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container"
Sep 16 18:58:43.721465 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721276    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kube-rbac-proxy"
Sep 16 18:58:43.721465 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.721285    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" containerName="kserve-container"
Sep 16 18:58:43.724522 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.724502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.726489 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.726472    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 18:58:43.726561 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.726487    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 16 18:58:43.735809 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.735789    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"]
Sep 16 18:58:43.824562 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.824530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.824751 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.824569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.824751 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.824626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02eb9d25-40d6-4aab-8390-779c786ecaa3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.824751 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.824678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.824751 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.824735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhqh\" (UniqueName: \"kubernetes.io/projected/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-api-access-6bhqh\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.925543 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.925512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.925695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.925551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.925695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.925578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02eb9d25-40d6-4aab-8390-779c786ecaa3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.925695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.925612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.925695 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.925675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhqh\" (UniqueName: \"kubernetes.io/projected/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-api-access-6bhqh\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.925695 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:43.925680    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:58:43.925911 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:43.925745    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls podName:02eb9d25-40d6-4aab-8390-779c786ecaa3 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:44.425724056 +0000 UTC m=+926.842426793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" (UID: "02eb9d25-40d6-4aab-8390-779c786ecaa3") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:58:43.926091 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.926058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02eb9d25-40d6-4aab-8390-779c786ecaa3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.926273 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.926258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.926337 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.926295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:43.933813 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:43.933788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhqh\" (UniqueName: \"kubernetes.io/projected/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-api-access-6bhqh\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:44.122425 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.122393    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be2163c7-d8d6-46a5-82bf-d1b4510b478f" path="/var/lib/kubelet/pods/be2163c7-d8d6-46a5-82bf-d1b4510b478f/volumes"
Sep 16 18:58:44.304309 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.304281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:44.353101 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.353072    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"]
Sep 16 18:58:44.358094 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.358073    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.360581 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.360559    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 16 18:58:44.360923 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.360908    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 16 18:58:44.368370 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.368343    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"]
Sep 16 18:58:44.372787 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.372729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:44.429506 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.429482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:44.432040 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.432014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:44.530236 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.530203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kserve-provision-location\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.530408 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.530250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5wj\" (UniqueName: \"kubernetes.io/projected/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-api-access-pr5wj\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.530408 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.530301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.530408 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.530352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.530517 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.530441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-cabundle-cert\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631273 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kserve-provision-location\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631273 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5wj\" (UniqueName: \"kubernetes.io/projected/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-api-access-pr5wj\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631273 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631587 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631587 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-cabundle-cert\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631587 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:44.631500    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 16 18:58:44.631587 ip-10-0-128-180 kubenswrapper[2562]: E0916 18:58:44.631578    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls podName:6185ecda-485a-4ee9-aaf2-1022ae6a04e0 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:45.131556218 +0000 UTC m=+927.548258956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls") pod "isvc-logger-predictor-6cb66b5bc5-h6kb7" (UID: "6185ecda-485a-4ee9-aaf2-1022ae6a04e0") : secret "isvc-logger-predictor-serving-cert" not found
Sep 16 18:58:44.631810 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kserve-provision-location\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.631975 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.631956    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-cabundle-cert\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.632042 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.632002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.634593 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.634570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:44.639784 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.639763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5wj\" (UniqueName: \"kubernetes.io/projected/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-api-access-pr5wj\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:44.688741 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.688708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:44.906644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:44.906621    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"]
Sep 16 18:58:44.908706 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:58:44.908673    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eb9d25_40d6_4aab_8390_779c786ecaa3.slice/crio-01cfbf641a02e387cf1e40dfef4e623036f8488d076a4bbdee4637211100220c WatchSource:0}: Error finding container 01cfbf641a02e387cf1e40dfef4e623036f8488d076a4bbdee4637211100220c: Status 404 returned error can't find the container with id 01cfbf641a02e387cf1e40dfef4e623036f8488d076a4bbdee4637211100220c
Sep 16 18:58:45.136918 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:45.136881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:45.139558 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:45.139534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls\") pod \"isvc-logger-predictor-6cb66b5bc5-h6kb7\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:45.266869 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:45.266765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerStarted","Data":"cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca"}
Sep 16 18:58:45.266869 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:45.266851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerStarted","Data":"01cfbf641a02e387cf1e40dfef4e623036f8488d076a4bbdee4637211100220c"}
Sep 16 18:58:45.269286 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:45.269263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:45.546727 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:45.546644    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"]
Sep 16 18:58:45.549627 ip-10-0-128-180 kubenswrapper[2562]: W0916 18:58:45.549603    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6185ecda_485a_4ee9_aaf2_1022ae6a04e0.slice/crio-0816129f1b7dff3a6dec83457e016dd533cb7f301bacbd5dfec82e5bda37c226 WatchSource:0}: Error finding container 0816129f1b7dff3a6dec83457e016dd533cb7f301bacbd5dfec82e5bda37c226: Status 404 returned error can't find the container with id 0816129f1b7dff3a6dec83457e016dd533cb7f301bacbd5dfec82e5bda37c226
Sep 16 18:58:46.272683 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:46.272645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerStarted","Data":"c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e"}
Sep 16 18:58:46.273122 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:46.272710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerStarted","Data":"0816129f1b7dff3a6dec83457e016dd533cb7f301bacbd5dfec82e5bda37c226"}
Sep 16 18:58:46.739131 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:46.739086    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:49.289460 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:49.289426    2562 generic.go:356] "Generic (PLEG): container finished" podID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerID="cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca" exitCode=0
Sep 16 18:58:49.289818 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:49.289499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerDied","Data":"cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca"}
Sep 16 18:58:50.297349 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:50.297298    2562 generic.go:356] "Generic (PLEG): container finished" podID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerID="c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e" exitCode=0
Sep 16 18:58:50.297778 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:50.297374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerDied","Data":"c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e"}
Sep 16 18:58:50.300156 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:50.300136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerStarted","Data":"253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c"}
Sep 16 18:58:50.300253 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:50.300189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerStarted","Data":"02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c"}
Sep 16 18:58:50.302582 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:50.302557    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:50.338598 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:50.338547    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podStartSLOduration=3.681577629 podStartE2EDuration="7.338525882s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.656948253s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:50.33790119 +0000 UTC m=+932.754603950" watchObservedRunningTime="2026-09-16 18:58:50.338525882 +0000 UTC m=+932.755228643"
Sep 16 18:58:51.309354 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.309304    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerStarted","Data":"9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6"}
Sep 16 18:58:51.309735 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.309390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerStarted","Data":"9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df"}
Sep 16 18:58:51.309735 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.309421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerStarted","Data":"6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee"}
Sep 16 18:58:51.309735 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.309575    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:51.309880 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.309856    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:51.310019 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.309999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:51.311171 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.311146    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:58:51.311171 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.311164    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:58:51.331439 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.331398    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podStartSLOduration=3.523157243 podStartE2EDuration="7.331383278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.808226035s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:51.32911602 +0000 UTC m=+933.745818789" watchObservedRunningTime="2026-09-16 18:58:51.331383278 +0000 UTC m=+933.748086037"
Sep 16 18:58:51.738857 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.738772    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:58:51.744203 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.744177    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 16 18:58:51.744367 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.744355    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:51.745653 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.745633    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:51.745747 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:51.745734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:52.314681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:52.314639    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:58:52.315120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:52.314640    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:58:52.315120 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:52.314737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:52.315613 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:52.315593    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:53.319200 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:53.319160    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:58:53.319631 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:53.319611    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:54.327053 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.327024    2562 generic.go:356] "Generic (PLEG): container finished" podID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerID="090f89c0c52a33d26d821aca5f571a147d2661d3707030514af205c615ec6ef6" exitCode=0
Sep 16 18:58:54.327412 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.327096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerDied","Data":"090f89c0c52a33d26d821aca5f571a147d2661d3707030514af205c615ec6ef6"}
Sep 16 18:58:54.378374 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.378352    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:54.425229 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425196    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zczdj\" (UniqueName: \"kubernetes.io/projected/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-api-access-zczdj\") pod \"e3e454ac-c9c0-4d63-b968-b5a04255da29\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") "
Sep 16 18:58:54.425382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425257    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e454ac-c9c0-4d63-b968-b5a04255da29-proxy-tls\") pod \"e3e454ac-c9c0-4d63-b968-b5a04255da29\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") "
Sep 16 18:58:54.425382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425279    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-cabundle-cert\") pod \"e3e454ac-c9c0-4d63-b968-b5a04255da29\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") "
Sep 16 18:58:54.425382 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425334    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-rbac-proxy-sar-config\") pod \"e3e454ac-c9c0-4d63-b968-b5a04255da29\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") "
Sep 16 18:58:54.425566 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e454ac-c9c0-4d63-b968-b5a04255da29-kserve-provision-location\") pod \"e3e454ac-c9c0-4d63-b968-b5a04255da29\" (UID: \"e3e454ac-c9c0-4d63-b968-b5a04255da29\") "
Sep 16 18:58:54.425790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425758    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3e454ac-c9c0-4d63-b968-b5a04255da29" (UID: "e3e454ac-c9c0-4d63-b968-b5a04255da29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:54.425790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e454ac-c9c0-4d63-b968-b5a04255da29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3e454ac-c9c0-4d63-b968-b5a04255da29" (UID: "e3e454ac-c9c0-4d63-b968-b5a04255da29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:54.425790 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3e454ac-c9c0-4d63-b968-b5a04255da29" (UID: "e3e454ac-c9c0-4d63-b968-b5a04255da29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:54.426215 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425830    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:54.426215 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425849    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:54.426215 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.425862    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e454ac-c9c0-4d63-b968-b5a04255da29-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:54.427534 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.427510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-api-access-zczdj" (OuterVolumeSpecName: "kube-api-access-zczdj") pod "e3e454ac-c9c0-4d63-b968-b5a04255da29" (UID: "e3e454ac-c9c0-4d63-b968-b5a04255da29"). InnerVolumeSpecName "kube-api-access-zczdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:54.427635 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.427550    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e454ac-c9c0-4d63-b968-b5a04255da29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3e454ac-c9c0-4d63-b968-b5a04255da29" (UID: "e3e454ac-c9c0-4d63-b968-b5a04255da29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:54.527116 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.527041    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zczdj\" (UniqueName: \"kubernetes.io/projected/e3e454ac-c9c0-4d63-b968-b5a04255da29-kube-api-access-zczdj\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:54.527116 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:54.527069    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e454ac-c9c0-4d63-b968-b5a04255da29-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 18:58:55.334844 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.334810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs" event={"ID":"e3e454ac-c9c0-4d63-b968-b5a04255da29","Type":"ContainerDied","Data":"eda34073d41e7982929f14891e32ac813afb125faa95d59937e2cee4d29a08f2"}
Sep 16 18:58:55.335258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.334862    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"
Sep 16 18:58:55.335258 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.334882    2562 scope.go:117] "RemoveContainer" containerID="090f89c0c52a33d26d821aca5f571a147d2661d3707030514af205c615ec6ef6"
Sep 16 18:58:55.343638 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.343623    2562 scope.go:117] "RemoveContainer" containerID="81efa70b18b66611ea57325babe2481bc8020115f954c5274578a4cdb85a2de6"
Sep 16 18:58:55.351264 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.351248    2562 scope.go:117] "RemoveContainer" containerID="1f08b32ad97cf3363e32e8a5f7820c31e47a2db22a0f9d946885fabc67f18224"
Sep 16 18:58:55.356113 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.356092    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"]
Sep 16 18:58:55.359602 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.359585    2562 scope.go:117] "RemoveContainer" containerID="cc0bc0034648ef6f28f78d34078733473459660dd57e840d5f77eeaeec3e8236"
Sep 16 18:58:55.360266 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:55.360249    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d9f644498-l6hgs"]
Sep 16 18:58:56.123023 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:56.122987    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" path="/var/lib/kubelet/pods/e3e454ac-c9c0-4d63-b968-b5a04255da29/volumes"
Sep 16 18:58:57.318866 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:57.318841    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 18:58:57.319276 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:57.319234    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:58:58.323733 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:58.323706    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 18:58:58.324296 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:58.324257    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:58:58.324613 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:58:58.324588    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:07.319529 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:07.319486    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:08.324205 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:08.324132    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:59:08.324652 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:08.324584    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:17.320078 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:17.320041    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:18.324152 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:18.324109    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:59:18.324644 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:18.324601    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:19.918579 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:19.918538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:27.320003 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:27.319954    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:28.324176 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:28.324131    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:59:28.324752 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:28.324723    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:29.303340 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:29.303290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:37.319513 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:37.319474    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:38.324185 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:38.324128    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:59:38.324661 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:38.324566    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:47.319818 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:47.319733    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:48.324174 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:48.324131    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:59:48.324681 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:48.324608    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:57.319773 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:57.319737    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 18:59:58.324200 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:58.324165    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 18:59:58.324775 ip-10-0-128-180 kubenswrapper[2562]: I0916 18:59:58.324743    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:02.117915 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:02.117864    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:02.118340 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:02.118208    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:07.320344 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:07.320291    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 19:00:12.122373 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:12.122313    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:12.122843 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:12.122454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:13.782290 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:13.782258    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"]
Sep 16 19:00:13.782787 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:13.782516    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" containerID="cri-o://02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c" gracePeriod=30
Sep 16 19:00:13.782787 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:13.782583    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kube-rbac-proxy" containerID="cri-o://253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c" gracePeriod=30
Sep 16 19:00:13.810479 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:13.810454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:14.703764 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:14.703735    2562 generic.go:356] "Generic (PLEG): container finished" podID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerID="253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c" exitCode=2
Sep 16 19:00:14.703932 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:14.703787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerDied","Data":"253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c"}
Sep 16 19:00:17.314948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:17.314904    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 16 19:00:17.320269 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:17.320232    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 16 19:00:18.994869 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:18.994839    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:19.153425 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.153397    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:19.333515 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.333484    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-nb6l2_d9f46fde-990b-4908-a9f9-162c9de371e8/kserve-container/0.log"
Sep 16 19:00:19.532538 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.532503    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"]
Sep 16 19:00:19.532845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.532818    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" containerID="cri-o://6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee" gracePeriod=30
Sep 16 19:00:19.532944 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.532907    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" containerID="cri-o://9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6" gracePeriod=30
Sep 16 19:00:19.532944 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.532922    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" containerID="cri-o://9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df" gracePeriod=30
Sep 16 19:00:19.553796 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.553765    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"]
Sep 16 19:00:19.554121 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.554074    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kserve-container" containerID="cri-o://ef6f008894899fde2800df9a8532737f35d99970b5a0e86deadfc7d9c46f93ce" gracePeriod=30
Sep 16 19:00:19.554235 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.554117    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kube-rbac-proxy" containerID="cri-o://9d211823285b507aaa5b7f63ce2a05ae653c948377e44482963156648101c332" gracePeriod=30
Sep 16 19:00:19.563103 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.563076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:19.664582 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.664555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"]
Sep 16 19:00:19.665047 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665022    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container"
Sep 16 19:00:19.665047 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665044    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container"
Sep 16 19:00:19.665232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665060    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent"
Sep 16 19:00:19.665232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665068    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent"
Sep 16 19:00:19.665232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665093    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy"
Sep 16 19:00:19.665232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665102    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy"
Sep 16 19:00:19.665232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="storage-initializer"
Sep 16 19:00:19.665232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665165    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="storage-initializer"
Sep 16 19:00:19.665695 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665303    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kserve-container"
Sep 16 19:00:19.665695 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665356    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="agent"
Sep 16 19:00:19.665695 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.665372    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e454ac-c9c0-4d63-b968-b5a04255da29" containerName="kube-rbac-proxy"
Sep 16 19:00:19.668911 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.668893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.669701 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.669682    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 19:00:19.671039 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.671019    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 16 19:00:19.671385 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.671362    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:00:19.679009 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.678987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"]
Sep 16 19:00:19.729207 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.729169    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerID="9d211823285b507aaa5b7f63ce2a05ae653c948377e44482963156648101c332" exitCode=2
Sep 16 19:00:19.729207 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.729206    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerID="ef6f008894899fde2800df9a8532737f35d99970b5a0e86deadfc7d9c46f93ce" exitCode=2
Sep 16 19:00:19.729442 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.729284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" event={"ID":"d9f46fde-990b-4908-a9f9-162c9de371e8","Type":"ContainerDied","Data":"9d211823285b507aaa5b7f63ce2a05ae653c948377e44482963156648101c332"}
Sep 16 19:00:19.729442 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.729356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" event={"ID":"d9f46fde-990b-4908-a9f9-162c9de371e8","Type":"ContainerDied","Data":"ef6f008894899fde2800df9a8532737f35d99970b5a0e86deadfc7d9c46f93ce"}
Sep 16 19:00:19.731926 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.731899    2562 generic.go:356] "Generic (PLEG): container finished" podID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerID="02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c" exitCode=0
Sep 16 19:00:19.732061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.731985    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"
Sep 16 19:00:19.732137 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.731985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerDied","Data":"02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c"}
Sep 16 19:00:19.732191 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.732139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt" event={"ID":"02eb9d25-40d6-4aab-8390-779c786ecaa3","Type":"ContainerDied","Data":"01cfbf641a02e387cf1e40dfef4e623036f8488d076a4bbdee4637211100220c"}
Sep 16 19:00:19.732191 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.732185    2562 scope.go:117] "RemoveContainer" containerID="253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c"
Sep 16 19:00:19.735215 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.735130    2562 generic.go:356] "Generic (PLEG): container finished" podID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerID="9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df" exitCode=2
Sep 16 19:00:19.735215 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.735182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerDied","Data":"9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df"}
Sep 16 19:00:19.744253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.744221    2562 scope.go:117] "RemoveContainer" containerID="02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c"
Sep 16 19:00:19.753663 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.753641    2562 scope.go:117] "RemoveContainer" containerID="cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca"
Sep 16 19:00:19.762253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.762228    2562 scope.go:117] "RemoveContainer" containerID="253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c"
Sep 16 19:00:19.762592 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:19.762575    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c\": container with ID starting with 253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c not found: ID does not exist" containerID="253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c"
Sep 16 19:00:19.762680 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.762605    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c"} err="failed to get container status \"253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c\": rpc error: code = NotFound desc = could not find container \"253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c\": container with ID starting with 253d79d8764e0874f6fc3631de1e2ba2d00834dd1fc6c2d80fa2b75a1c81049c not found: ID does not exist"
Sep 16 19:00:19.762680 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.762628    2562 scope.go:117] "RemoveContainer" containerID="02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c"
Sep 16 19:00:19.762909 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:19.762892    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c\": container with ID starting with 02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c not found: ID does not exist" containerID="02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c"
Sep 16 19:00:19.762950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.762915    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c"} err="failed to get container status \"02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c\": rpc error: code = NotFound desc = could not find container \"02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c\": container with ID starting with 02ecca679f5c436326e897537b363671ba5ec676d1d25584dc3672ae2d54606c not found: ID does not exist"
Sep 16 19:00:19.762950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.762929    2562 scope.go:117] "RemoveContainer" containerID="cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca"
Sep 16 19:00:19.763194 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:19.763178    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca\": container with ID starting with cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca not found: ID does not exist" containerID="cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca"
Sep 16 19:00:19.763245 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.763201    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca"} err="failed to get container status \"cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca\": rpc error: code = NotFound desc = could not find container \"cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca\": container with ID starting with cfe8eaab29c91173b761a1ffd9d5ee0045cc4040168c2f123684ddbb188d2bca not found: ID does not exist"
Sep 16 19:00:19.766889 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.766868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bhqh\" (UniqueName: \"kubernetes.io/projected/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-api-access-6bhqh\") pod \"02eb9d25-40d6-4aab-8390-779c786ecaa3\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") "
Sep 16 19:00:19.766989 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.766906    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls\") pod \"02eb9d25-40d6-4aab-8390-779c786ecaa3\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") "
Sep 16 19:00:19.766989 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.766934    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-cabundle-cert\") pod \"02eb9d25-40d6-4aab-8390-779c786ecaa3\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") "
Sep 16 19:00:19.766989 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.766959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-rbac-proxy-sar-config\") pod \"02eb9d25-40d6-4aab-8390-779c786ecaa3\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") "
Sep 16 19:00:19.767161 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02eb9d25-40d6-4aab-8390-779c786ecaa3-kserve-provision-location\") pod \"02eb9d25-40d6-4aab-8390-779c786ecaa3\" (UID: \"02eb9d25-40d6-4aab-8390-779c786ecaa3\") "
Sep 16 19:00:19.767161 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/713fa52b-c4af-4773-9273-6c2954edc49b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.767375 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.767375 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767283    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02eb9d25-40d6-4aab-8390-779c786ecaa3" (UID: "02eb9d25-40d6-4aab-8390-779c786ecaa3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:19.767375 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/713fa52b-c4af-4773-9273-6c2954edc49b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.767534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhkck\" (UniqueName: \"kubernetes.io/projected/713fa52b-c4af-4773-9273-6c2954edc49b-kube-api-access-nhkck\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.767534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.767534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767495    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:19.767709 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767663    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02eb9d25-40d6-4aab-8390-779c786ecaa3" (UID: "02eb9d25-40d6-4aab-8390-779c786ecaa3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:19.767770 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.767752    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb9d25-40d6-4aab-8390-779c786ecaa3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02eb9d25-40d6-4aab-8390-779c786ecaa3" (UID: "02eb9d25-40d6-4aab-8390-779c786ecaa3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:19.768990 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.768969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-api-access-6bhqh" (OuterVolumeSpecName: "kube-api-access-6bhqh") pod "02eb9d25-40d6-4aab-8390-779c786ecaa3" (UID: "02eb9d25-40d6-4aab-8390-779c786ecaa3"). InnerVolumeSpecName "kube-api-access-6bhqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:19.769102 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.769094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02eb9d25-40d6-4aab-8390-779c786ecaa3" (UID: "02eb9d25-40d6-4aab-8390-779c786ecaa3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:19.805716 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.805695    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 19:00:19.868036 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868172 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/713fa52b-c4af-4773-9273-6c2954edc49b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868172 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhkck\" (UniqueName: \"kubernetes.io/projected/713fa52b-c4af-4773-9273-6c2954edc49b-kube-api-access-nhkck\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868172 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868172 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/713fa52b-c4af-4773-9273-6c2954edc49b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868418 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868179    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhqh\" (UniqueName: \"kubernetes.io/projected/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-api-access-6bhqh\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:19.868418 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868192    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02eb9d25-40d6-4aab-8390-779c786ecaa3-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:19.868418 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868204    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02eb9d25-40d6-4aab-8390-779c786ecaa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:19.868418 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868217    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02eb9d25-40d6-4aab-8390-779c786ecaa3-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:19.868571 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/713fa52b-c4af-4773-9273-6c2954edc49b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868771 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868753    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.868843 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.868825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.870661 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.870646    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/713fa52b-c4af-4773-9273-6c2954edc49b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.876528 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.876511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhkck\" (UniqueName: \"kubernetes.io/projected/713fa52b-c4af-4773-9273-6c2954edc49b-kube-api-access-nhkck\") pod \"isvc-autogluon-v1-predictor-574b5b7495-png7f\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:19.965278 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.965199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:19.969287 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.969264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-rbac-proxy-sar-config\") pod \"d9f46fde-990b-4908-a9f9-162c9de371e8\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") "
Sep 16 19:00:19.969441 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.969315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f46fde-990b-4908-a9f9-162c9de371e8-proxy-tls\") pod \"d9f46fde-990b-4908-a9f9-162c9de371e8\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") "
Sep 16 19:00:19.969441 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.969371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjvhq\" (UniqueName: \"kubernetes.io/projected/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-api-access-jjvhq\") pod \"d9f46fde-990b-4908-a9f9-162c9de371e8\" (UID: \"d9f46fde-990b-4908-a9f9-162c9de371e8\") "
Sep 16 19:00:19.969674 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.969652    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9f46fde-990b-4908-a9f9-162c9de371e8" (UID: "d9f46fde-990b-4908-a9f9-162c9de371e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:19.972785 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.972734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-api-access-jjvhq" (OuterVolumeSpecName: "kube-api-access-jjvhq") pod "d9f46fde-990b-4908-a9f9-162c9de371e8" (UID: "d9f46fde-990b-4908-a9f9-162c9de371e8"). InnerVolumeSpecName "kube-api-access-jjvhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:19.977402 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.977379    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f46fde-990b-4908-a9f9-162c9de371e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9f46fde-990b-4908-a9f9-162c9de371e8" (UID: "d9f46fde-990b-4908-a9f9-162c9de371e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:19.980508 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:19.980487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:00:20.044239 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.044203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:20.057521 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.057495    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"]
Sep 16 19:00:20.062394 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.062371    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t9hvt"]
Sep 16 19:00:20.070397 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.070378    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f46fde-990b-4908-a9f9-162c9de371e8-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:20.070397 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.070398    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvhq\" (UniqueName: \"kubernetes.io/projected/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-api-access-jjvhq\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:20.070533 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.070408    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f46fde-990b-4908-a9f9-162c9de371e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:20.122822 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.122799    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" path="/var/lib/kubelet/pods/02eb9d25-40d6-4aab-8390-779c786ecaa3/volumes"
Sep 16 19:00:20.254110 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.254084    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"]
Sep 16 19:00:20.256998 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:00:20.256971    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713fa52b_c4af_4773_9273_6c2954edc49b.slice/crio-8cf50f061bc1493175673d79a31732eeab2ecd58ff655dc456da58814bf467d3 WatchSource:0}: Error finding container 8cf50f061bc1493175673d79a31732eeab2ecd58ff655dc456da58814bf467d3: Status 404 returned error can't find the container with id 8cf50f061bc1493175673d79a31732eeab2ecd58ff655dc456da58814bf467d3
Sep 16 19:00:20.742185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.742153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2" event={"ID":"d9f46fde-990b-4908-a9f9-162c9de371e8","Type":"ContainerDied","Data":"e6e54ab69c00ee0bd1bdd16eeb209689c2d87e97c074c5fdaf61c7dc2fbdfe49"}
Sep 16 19:00:20.742393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.742223    2562 scope.go:117] "RemoveContainer" containerID="9d211823285b507aaa5b7f63ce2a05ae653c948377e44482963156648101c332"
Sep 16 19:00:20.742393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.742162    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"
Sep 16 19:00:20.745379 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.745352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerStarted","Data":"80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52"}
Sep 16 19:00:20.745472 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.745412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerStarted","Data":"8cf50f061bc1493175673d79a31732eeab2ecd58ff655dc456da58814bf467d3"}
Sep 16 19:00:20.750772 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.750754    2562 scope.go:117] "RemoveContainer" containerID="ef6f008894899fde2800df9a8532737f35d99970b5a0e86deadfc7d9c46f93ce"
Sep 16 19:00:20.778008 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.777981    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"]
Sep 16 19:00:20.781588 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:20.781568    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nb6l2"]
Sep 16 19:00:22.117939 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:22.117897    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:22.118441 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:22.118234    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:22.121703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:22.121681    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" path="/var/lib/kubelet/pods/d9f46fde-990b-4908-a9f9-162c9de371e8/volumes"
Sep 16 19:00:23.320146 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:23.320103    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:00:24.768754 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:24.768715    2562 generic.go:356] "Generic (PLEG): container finished" podID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerID="6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee" exitCode=0
Sep 16 19:00:24.769091 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:24.768792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerDied","Data":"6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee"}
Sep 16 19:00:25.774760 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:25.774726    2562 generic.go:356] "Generic (PLEG): container finished" podID="713fa52b-c4af-4773-9273-6c2954edc49b" containerID="80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52" exitCode=0
Sep 16 19:00:25.775150 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:25.774805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerDied","Data":"80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52"}
Sep 16 19:00:28.319638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:28.319596    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:00:32.118416 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:32.118358    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:32.118958 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:32.118787    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:33.320067 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:33.320030    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:00:33.320470 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:33.320173    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:38.319756 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:38.319713    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:00:42.117928 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:42.117872    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 16 19:00:42.118340 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:42.118217    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:42.122248 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:42.122225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:42.122379 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:42.122343    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:43.319875 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:43.319842    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:00:48.319940 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:48.319897    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 16 19:00:49.702074 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.702039    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:49.832787 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.832754    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kserve-provision-location\") pod \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") "
Sep 16 19:00:49.832787 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.832794    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr5wj\" (UniqueName: \"kubernetes.io/projected/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-api-access-pr5wj\") pod \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") "
Sep 16 19:00:49.833041 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.832879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-rbac-proxy-sar-config\") pod \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") "
Sep 16 19:00:49.833041 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.832906    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-cabundle-cert\") pod \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") "
Sep 16 19:00:49.833041 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.832943    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls\") pod \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\" (UID: \"6185ecda-485a-4ee9-aaf2-1022ae6a04e0\") "
Sep 16 19:00:49.833190 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.833067    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6185ecda-485a-4ee9-aaf2-1022ae6a04e0" (UID: "6185ecda-485a-4ee9-aaf2-1022ae6a04e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:49.833237 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.833194    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:49.833574 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.833537    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6185ecda-485a-4ee9-aaf2-1022ae6a04e0" (UID: "6185ecda-485a-4ee9-aaf2-1022ae6a04e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:49.833574 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.833556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6185ecda-485a-4ee9-aaf2-1022ae6a04e0" (UID: "6185ecda-485a-4ee9-aaf2-1022ae6a04e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:49.835515 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.835492    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6185ecda-485a-4ee9-aaf2-1022ae6a04e0" (UID: "6185ecda-485a-4ee9-aaf2-1022ae6a04e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:49.835829 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.835807    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-api-access-pr5wj" (OuterVolumeSpecName: "kube-api-access-pr5wj") pod "6185ecda-485a-4ee9-aaf2-1022ae6a04e0" (UID: "6185ecda-485a-4ee9-aaf2-1022ae6a04e0"). InnerVolumeSpecName "kube-api-access-pr5wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:49.909197 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.909157    2562 generic.go:356] "Generic (PLEG): container finished" podID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerID="9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6" exitCode=137
Sep 16 19:00:49.909399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.909238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"
Sep 16 19:00:49.909399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.909252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerDied","Data":"9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6"}
Sep 16 19:00:49.909399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.909351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7" event={"ID":"6185ecda-485a-4ee9-aaf2-1022ae6a04e0","Type":"ContainerDied","Data":"0816129f1b7dff3a6dec83457e016dd533cb7f301bacbd5dfec82e5bda37c226"}
Sep 16 19:00:49.909399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.909388    2562 scope.go:117] "RemoveContainer" containerID="9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6"
Sep 16 19:00:49.920241 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.920215    2562 scope.go:117] "RemoveContainer" containerID="9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df"
Sep 16 19:00:49.930414 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.930389    2562 scope.go:117] "RemoveContainer" containerID="6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee"
Sep 16 19:00:49.933876 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.933852    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:49.933969 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.933879    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:49.933969 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.933894    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:49.933969 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.933905    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5wj\" (UniqueName: \"kubernetes.io/projected/6185ecda-485a-4ee9-aaf2-1022ae6a04e0-kube-api-access-pr5wj\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:00:49.940763 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.940742    2562 scope.go:117] "RemoveContainer" containerID="c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e"
Sep 16 19:00:49.940858 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.940772    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"]
Sep 16 19:00:49.944904 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.944881    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6cb66b5bc5-h6kb7"]
Sep 16 19:00:49.950907 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.950890    2562 scope.go:117] "RemoveContainer" containerID="9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6"
Sep 16 19:00:49.951202 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:49.951184    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6\": container with ID starting with 9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6 not found: ID does not exist" containerID="9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6"
Sep 16 19:00:49.951253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951212    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6"} err="failed to get container status \"9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6\": rpc error: code = NotFound desc = could not find container \"9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6\": container with ID starting with 9d357b08de7e4b68ac196607bf40cd61dac1c0a83d4d27e0264f8cccea7271b6 not found: ID does not exist"
Sep 16 19:00:49.951253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951229    2562 scope.go:117] "RemoveContainer" containerID="9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df"
Sep 16 19:00:49.951495 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:49.951478    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df\": container with ID starting with 9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df not found: ID does not exist" containerID="9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df"
Sep 16 19:00:49.951551 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951504    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df"} err="failed to get container status \"9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df\": rpc error: code = NotFound desc = could not find container \"9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df\": container with ID starting with 9f9984ff7ec4825bb68d01a9b7846e9b7834f0812bfbec3f505efdd3950376df not found: ID does not exist"
Sep 16 19:00:49.951551 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951523    2562 scope.go:117] "RemoveContainer" containerID="6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee"
Sep 16 19:00:49.951736 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:49.951721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee\": container with ID starting with 6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee not found: ID does not exist" containerID="6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee"
Sep 16 19:00:49.951778 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951741    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee"} err="failed to get container status \"6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee\": rpc error: code = NotFound desc = could not find container \"6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee\": container with ID starting with 6da9f80c89cba906239a48e14022431bccc1ffd93bc842acbc12a859b41f25ee not found: ID does not exist"
Sep 16 19:00:49.951778 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951754    2562 scope.go:117] "RemoveContainer" containerID="c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e"
Sep 16 19:00:49.951952 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:00:49.951934    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e\": container with ID starting with c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e not found: ID does not exist" containerID="c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e"
Sep 16 19:00:49.951993 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:49.951955    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e"} err="failed to get container status \"c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e\": rpc error: code = NotFound desc = could not find container \"c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e\": container with ID starting with c9c26fd0e53781f9be48e01a794b29e4896c9b6c8d62b2b13e443ae6620e629e not found: ID does not exist"
Sep 16 19:00:50.122624 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:00:50.122544    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" path="/var/lib/kubelet/pods/6185ecda-485a-4ee9-aaf2-1022ae6a04e0/volumes"
Sep 16 19:02:19.301026 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:19.300984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:23.375018 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:23.374978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerStarted","Data":"d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815"}
Sep 16 19:02:23.375622 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:23.375037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerStarted","Data":"fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e"}
Sep 16 19:02:23.375622 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:23.375135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:02:23.411430 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:23.411372    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" podStartSLOduration=3.120723101 podStartE2EDuration="2m4.411356323s" totalImagesPullingTime="1m56.631183155s" totalInitContainerRuntime="4.659450067s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:23.40713108 +0000 UTC m=+1145.823833843" watchObservedRunningTime="2026-09-16 19:02:23.411356323 +0000 UTC m=+1145.828059101"
Sep 16 19:02:24.289508 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:24.289467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:24.379649 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:24.379614    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:02:24.380985 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:24.380956    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:02:24.618024 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:24.617994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:25.385278 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:25.385243    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:02:30.389398 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:30.389367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:02:30.389855 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:30.389819    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:02:39.161214 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:39.161183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:39.242444 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:39.242408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:40.390508 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:40.390441    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:02:55.895906 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:55.895871    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"]
Sep 16 19:02:55.896473 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:55.896128    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container" containerID="cri-o://fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e" gracePeriod=30
Sep 16 19:02:55.896473 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:55.896172    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kube-rbac-proxy" containerID="cri-o://d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815" gracePeriod=30
Sep 16 19:02:55.978992 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:55.978945    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:56.521436 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:56.521396    2562 generic.go:356] "Generic (PLEG): container finished" podID="713fa52b-c4af-4773-9273-6c2954edc49b" containerID="d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815" exitCode=2
Sep 16 19:02:56.521637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:56.521446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerDied","Data":"d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815"}
Sep 16 19:02:57.342561 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.342528    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:02:57.386311 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386233    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/713fa52b-c4af-4773-9273-6c2954edc49b-kserve-provision-location\") pod \"713fa52b-c4af-4773-9273-6c2954edc49b\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") "
Sep 16 19:02:57.386311 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-kube-rbac-proxy-sar-config\") pod \"713fa52b-c4af-4773-9273-6c2954edc49b\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") "
Sep 16 19:02:57.386534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhkck\" (UniqueName: \"kubernetes.io/projected/713fa52b-c4af-4773-9273-6c2954edc49b-kube-api-access-nhkck\") pod \"713fa52b-c4af-4773-9273-6c2954edc49b\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") "
Sep 16 19:02:57.386534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/713fa52b-c4af-4773-9273-6c2954edc49b-proxy-tls\") pod \"713fa52b-c4af-4773-9273-6c2954edc49b\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") "
Sep 16 19:02:57.386534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386398    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-cabundle-cert\") pod \"713fa52b-c4af-4773-9273-6c2954edc49b\" (UID: \"713fa52b-c4af-4773-9273-6c2954edc49b\") "
Sep 16 19:02:57.386781 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386758    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "713fa52b-c4af-4773-9273-6c2954edc49b" (UID: "713fa52b-c4af-4773-9273-6c2954edc49b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:57.386851 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.386783    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "713fa52b-c4af-4773-9273-6c2954edc49b" (UID: "713fa52b-c4af-4773-9273-6c2954edc49b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:57.388623 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.388599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713fa52b-c4af-4773-9273-6c2954edc49b-kube-api-access-nhkck" (OuterVolumeSpecName: "kube-api-access-nhkck") pod "713fa52b-c4af-4773-9273-6c2954edc49b" (UID: "713fa52b-c4af-4773-9273-6c2954edc49b"). InnerVolumeSpecName "kube-api-access-nhkck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:57.388739 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.388643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713fa52b-c4af-4773-9273-6c2954edc49b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "713fa52b-c4af-4773-9273-6c2954edc49b" (UID: "713fa52b-c4af-4773-9273-6c2954edc49b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:57.390638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.390614    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/713fa52b-c4af-4773-9273-6c2954edc49b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "713fa52b-c4af-4773-9273-6c2954edc49b" (UID: "713fa52b-c4af-4773-9273-6c2954edc49b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:57.487865 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.487825    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/713fa52b-c4af-4773-9273-6c2954edc49b-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:02:57.487865 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.487858    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:02:57.487865 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.487868    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhkck\" (UniqueName: \"kubernetes.io/projected/713fa52b-c4af-4773-9273-6c2954edc49b-kube-api-access-nhkck\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:02:57.488108 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.487877    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/713fa52b-c4af-4773-9273-6c2954edc49b-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:02:57.488108 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.487888    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/713fa52b-c4af-4773-9273-6c2954edc49b-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:02:57.527891 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.527858    2562 generic.go:356] "Generic (PLEG): container finished" podID="713fa52b-c4af-4773-9273-6c2954edc49b" containerID="fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e" exitCode=0
Sep 16 19:02:57.528046 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.527909    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"
Sep 16 19:02:57.528046 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.527943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerDied","Data":"fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e"}
Sep 16 19:02:57.528046 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.528015    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f" event={"ID":"713fa52b-c4af-4773-9273-6c2954edc49b","Type":"ContainerDied","Data":"8cf50f061bc1493175673d79a31732eeab2ecd58ff655dc456da58814bf467d3"}
Sep 16 19:02:57.528046 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.528043    2562 scope.go:117] "RemoveContainer" containerID="d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815"
Sep 16 19:02:57.537736 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.537721    2562 scope.go:117] "RemoveContainer" containerID="fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e"
Sep 16 19:02:57.545527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.545508    2562 scope.go:117] "RemoveContainer" containerID="80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52"
Sep 16 19:02:57.550772 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.550743    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"]
Sep 16 19:02:57.554229 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.554212    2562 scope.go:117] "RemoveContainer" containerID="d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815"
Sep 16 19:02:57.554551 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:02:57.554528    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815\": container with ID starting with d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815 not found: ID does not exist" containerID="d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815"
Sep 16 19:02:57.554606 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.554566    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815"} err="failed to get container status \"d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815\": rpc error: code = NotFound desc = could not find container \"d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815\": container with ID starting with d438330c58c28d93b5d622b828bfeb84908aac04a5bffab19bc69f9bd836f815 not found: ID does not exist"
Sep 16 19:02:57.554606 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.554591    2562 scope.go:117] "RemoveContainer" containerID="fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e"
Sep 16 19:02:57.554878 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:02:57.554860    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e\": container with ID starting with fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e not found: ID does not exist" containerID="fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e"
Sep 16 19:02:57.554924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.554886    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e"} err="failed to get container status \"fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e\": rpc error: code = NotFound desc = could not find container \"fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e\": container with ID starting with fec1c449be38d812272245b061de5ab6baa90cc0ff266619f0efa88bb1ea0a0e not found: ID does not exist"
Sep 16 19:02:57.554924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.554902    2562 scope.go:117] "RemoveContainer" containerID="80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52"
Sep 16 19:02:57.555039 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.554992    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-png7f"]
Sep 16 19:02:57.555128 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:02:57.555113    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52\": container with ID starting with 80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52 not found: ID does not exist" containerID="80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52"
Sep 16 19:02:57.555168 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:57.555132    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52"} err="failed to get container status \"80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52\": rpc error: code = NotFound desc = could not find container \"80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52\": container with ID starting with 80458d5772866e213fd3c5a1562b73fbf52627bb85e448e492084c1dea124d52 not found: ID does not exist"
Sep 16 19:02:58.122094 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:02:58.122059    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" path="/var/lib/kubelet/pods/713fa52b-c4af-4773-9273-6c2954edc49b/volumes"
Sep 16 19:03:01.134774 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:01.134743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:01.616564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:01.616533    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:54.395075 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:54.395024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:59.542559 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.542525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:59.569381 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569352    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"]
Sep 16 19:03:59.569663 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569652    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="storage-initializer"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569665    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="storage-initializer"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569673    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569679    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569689    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kserve-container"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569694    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kserve-container"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569700    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container"
Sep 16 19:03:59.569707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569705    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569714    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569719    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569728    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="storage-initializer"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569733    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="storage-initializer"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569739    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569743    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569752    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569757    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569763    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569768    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569776    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="storage-initializer"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569782    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="storage-initializer"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569786    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569790    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569801    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569846    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569854    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569859    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569865    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="agent"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569870    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02eb9d25-40d6-4aab-8390-779c786ecaa3" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569876    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569881    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6185ecda-485a-4ee9-aaf2-1022ae6a04e0" containerName="kube-rbac-proxy"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569887    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="713fa52b-c4af-4773-9273-6c2954edc49b" containerName="kserve-container"
Sep 16 19:03:59.569918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.569891    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f46fde-990b-4908-a9f9-162c9de371e8" containerName="kube-rbac-proxy"
Sep 16 19:03:59.572072 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.572056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.574683 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.574662    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:03:59.574791 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.574692    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:59.574981 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.574962    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:59.575079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.574999    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:59.575537 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.575517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:03:59.575628 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.575545    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:59.575628 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.575568    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 16 19:03:59.583061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.583039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"]
Sep 16 19:03:59.593575 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.593555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4nwc\" (UniqueName: \"kubernetes.io/projected/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-api-access-t4nwc\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.593661 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.593589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.593661 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.593657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.593749 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.593691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b2dbca-372a-4054-b099-5bfd7ac7d02f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.593749 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.593720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.639799 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.639721    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:59.695005 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.694974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b2dbca-372a-4054-b099-5bfd7ac7d02f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nwc\" (UniqueName: \"kubernetes.io/projected/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-api-access-t4nwc\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695537 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695781 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.695853 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.695814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.697745 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.697729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b2dbca-372a-4054-b099-5bfd7ac7d02f-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.704582 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.704563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nwc\" (UniqueName: \"kubernetes.io/projected/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-api-access-t4nwc\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:03:59.882881 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:03:59.882835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:04:00.156694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:00.156669    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"]
Sep 16 19:04:00.159097 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:04:00.159064    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b2dbca_372a_4054_b099_5bfd7ac7d02f.slice/crio-7e00f7fcdba92745b4ad02781337c972d307ba2bf8fb74986662081cd89b1739 WatchSource:0}: Error finding container 7e00f7fcdba92745b4ad02781337c972d307ba2bf8fb74986662081cd89b1739: Status 404 returned error can't find the container with id 7e00f7fcdba92745b4ad02781337c972d307ba2bf8fb74986662081cd89b1739
Sep 16 19:04:00.796761 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:00.796724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerStarted","Data":"b2561e11d27c17fb7ebd7f7f27e4e887cc4f491461f2a809cd6dffa97ab06cfc"}
Sep 16 19:04:00.797132 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:00.796783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerStarted","Data":"7e00f7fcdba92745b4ad02781337c972d307ba2bf8fb74986662081cd89b1739"}
Sep 16 19:04:04.814899 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:04.814867    2562 generic.go:356] "Generic (PLEG): container finished" podID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerID="b2561e11d27c17fb7ebd7f7f27e4e887cc4f491461f2a809cd6dffa97ab06cfc" exitCode=0
Sep 16 19:04:04.814899 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:04.814903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerDied","Data":"b2561e11d27c17fb7ebd7f7f27e4e887cc4f491461f2a809cd6dffa97ab06cfc"}
Sep 16 19:04:05.822122 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:05.822092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerStarted","Data":"f1ef97c14f680ab4429e413662a2b41c7dde9943c2013146dff7c2778e653661"}
Sep 16 19:04:05.822559 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:05.822156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerStarted","Data":"655a08db22a3a69e3d4109bd779c053c6420134f56a02fd0a7cfa00faf316c0d"}
Sep 16 19:04:05.822559 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:05.822392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:04:05.822649 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:05.822560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:04:05.842770 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:05.842727    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podStartSLOduration=3.183290263 podStartE2EDuration="6.842713339s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659423076s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:05.840392464 +0000 UTC m=+1248.257095223" watchObservedRunningTime="2026-09-16 19:04:05.842713339 +0000 UTC m=+1248.259416098"
Sep 16 19:04:11.832439 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:11.832368    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:04:23.409185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:23.409145    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:34.640018 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:34.639981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:41.833272 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:41.833231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:04:51.833779 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:04:51.833734    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:01.833751 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:01.833703    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:11.832989 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:11.832950    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:21.900947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:21.900917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:05:29.705876 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:29.705843    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"]
Sep 16 19:05:29.706365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:29.706137    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container" containerID="cri-o://655a08db22a3a69e3d4109bd779c053c6420134f56a02fd0a7cfa00faf316c0d" gracePeriod=30
Sep 16 19:05:29.706365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:29.706157    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kube-rbac-proxy" containerID="cri-o://f1ef97c14f680ab4429e413662a2b41c7dde9943c2013146dff7c2778e653661" gracePeriod=30
Sep 16 19:05:29.738524 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:29.738497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:30.175595 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:30.175558    2562 generic.go:356] "Generic (PLEG): container finished" podID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerID="f1ef97c14f680ab4429e413662a2b41c7dde9943c2013146dff7c2778e653661" exitCode=2
Sep 16 19:05:30.175766 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:30.175629    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerDied","Data":"f1ef97c14f680ab4429e413662a2b41c7dde9943c2013146dff7c2778e653661"}
Sep 16 19:05:31.826654 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:31.826616    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:05:31.833656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:31.833617    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:05:34.816099 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:34.816060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:35.045066 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.044982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:35.197608 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.197575    2562 generic.go:356] "Generic (PLEG): container finished" podID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerID="655a08db22a3a69e3d4109bd779c053c6420134f56a02fd0a7cfa00faf316c0d" exitCode=0
Sep 16 19:05:35.197608 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.197612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerDied","Data":"655a08db22a3a69e3d4109bd779c053c6420134f56a02fd0a7cfa00faf316c0d"}
Sep 16 19:05:35.334509 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.334489    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:05:35.419873 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.419841    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4nwc\" (UniqueName: \"kubernetes.io/projected/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-api-access-t4nwc\") pod \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") "
Sep 16 19:05:35.420028 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.419913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-cabundle-cert\") pod \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") "
Sep 16 19:05:35.420081 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b2dbca-372a-4054-b099-5bfd7ac7d02f-proxy-tls\") pod \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") "
Sep 16 19:05:35.420129 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420105    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-rbac-proxy-sar-config\") pod \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") "
Sep 16 19:05:35.420185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420149    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kserve-provision-location\") pod \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\" (UID: \"00b2dbca-372a-4054-b099-5bfd7ac7d02f\") "
Sep 16 19:05:35.420260 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420239    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00b2dbca-372a-4054-b099-5bfd7ac7d02f" (UID: "00b2dbca-372a-4054-b099-5bfd7ac7d02f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:35.420439 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420420    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:05:35.420551 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420527    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00b2dbca-372a-4054-b099-5bfd7ac7d02f" (UID: "00b2dbca-372a-4054-b099-5bfd7ac7d02f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:35.420640 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.420572    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00b2dbca-372a-4054-b099-5bfd7ac7d02f" (UID: "00b2dbca-372a-4054-b099-5bfd7ac7d02f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:35.422185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.422166    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-api-access-t4nwc" (OuterVolumeSpecName: "kube-api-access-t4nwc") pod "00b2dbca-372a-4054-b099-5bfd7ac7d02f" (UID: "00b2dbca-372a-4054-b099-5bfd7ac7d02f"). InnerVolumeSpecName "kube-api-access-t4nwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:35.422297 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.422249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b2dbca-372a-4054-b099-5bfd7ac7d02f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00b2dbca-372a-4054-b099-5bfd7ac7d02f" (UID: "00b2dbca-372a-4054-b099-5bfd7ac7d02f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:35.521727 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.521698    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4nwc\" (UniqueName: \"kubernetes.io/projected/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-api-access-t4nwc\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:05:35.521727 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.521724    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00b2dbca-372a-4054-b099-5bfd7ac7d02f-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:05:35.521874 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.521733    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:05:35.521874 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:35.521741    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00b2dbca-372a-4054-b099-5bfd7ac7d02f-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:05:36.203712 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.203680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t" event={"ID":"00b2dbca-372a-4054-b099-5bfd7ac7d02f","Type":"ContainerDied","Data":"7e00f7fcdba92745b4ad02781337c972d307ba2bf8fb74986662081cd89b1739"}
Sep 16 19:05:36.203712 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.203698    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"
Sep 16 19:05:36.204154 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.203770    2562 scope.go:117] "RemoveContainer" containerID="f1ef97c14f680ab4429e413662a2b41c7dde9943c2013146dff7c2778e653661"
Sep 16 19:05:36.212113 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.212097    2562 scope.go:117] "RemoveContainer" containerID="655a08db22a3a69e3d4109bd779c053c6420134f56a02fd0a7cfa00faf316c0d"
Sep 16 19:05:36.219266 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.219085    2562 scope.go:117] "RemoveContainer" containerID="b2561e11d27c17fb7ebd7f7f27e4e887cc4f491461f2a809cd6dffa97ab06cfc"
Sep 16 19:05:36.221985 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.221960    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"]
Sep 16 19:05:36.224259 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.224238    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b6f5t"]
Sep 16 19:05:36.860271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:36.860243    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:38.122827 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:38.122795    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" path="/var/lib/kubelet/pods/00b2dbca-372a-4054-b099-5bfd7ac7d02f/volumes"
Sep 16 19:05:52.421392 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421299    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-f8201add"]
Sep 16 19:05:52.421721 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421684    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="storage-initializer"
Sep 16 19:05:52.421721 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421698    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="storage-initializer"
Sep 16 19:05:52.421721 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421716    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container"
Sep 16 19:05:52.421721 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421721    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container"
Sep 16 19:05:52.421864 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421732    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kube-rbac-proxy"
Sep 16 19:05:52.421864 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421737    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kube-rbac-proxy"
Sep 16 19:05:52.421864 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421787    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kube-rbac-proxy"
Sep 16 19:05:52.421864 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.421798    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b2dbca-372a-4054-b099-5bfd7ac7d02f" containerName="kserve-container"
Sep 16 19:05:52.423605 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.423589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:52.425884 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.425862    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 19:05:52.426452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.426436    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 19:05:52.426543 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.426439    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 19:05:52.429196 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.429160    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-f8201add"]
Sep 16 19:05:52.566387 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.566352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqfh\" (UniqueName: \"kubernetes.io/projected/055ee9ef-e844-43ce-970c-359b5bebb918-kube-api-access-bnqfh\") pod \"curl-metrics-f8201add\" (UID: \"055ee9ef-e844-43ce-970c-359b5bebb918\") " pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:52.667632 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.667599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqfh\" (UniqueName: \"kubernetes.io/projected/055ee9ef-e844-43ce-970c-359b5bebb918-kube-api-access-bnqfh\") pod \"curl-metrics-f8201add\" (UID: \"055ee9ef-e844-43ce-970c-359b5bebb918\") " pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:52.676242 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.676175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqfh\" (UniqueName: \"kubernetes.io/projected/055ee9ef-e844-43ce-970c-359b5bebb918-kube-api-access-bnqfh\") pod \"curl-metrics-f8201add\" (UID: \"055ee9ef-e844-43ce-970c-359b5bebb918\") " pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:52.733200 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.733174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:52.999305 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:52.999144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-f8201add"]
Sep 16 19:05:53.001893 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:05:53.001862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod055ee9ef_e844_43ce_970c_359b5bebb918.slice/crio-57db5c27bf09570458a17fdbae989e36b66a32d9ba00f80a06572ae4706d3f42 WatchSource:0}: Error finding container 57db5c27bf09570458a17fdbae989e36b66a32d9ba00f80a06572ae4706d3f42: Status 404 returned error can't find the container with id 57db5c27bf09570458a17fdbae989e36b66a32d9ba00f80a06572ae4706d3f42
Sep 16 19:05:53.003849 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:53.003832    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:05:53.276505 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:53.276420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-f8201add" event={"ID":"055ee9ef-e844-43ce-970c-359b5bebb918","Type":"ContainerStarted","Data":"57db5c27bf09570458a17fdbae989e36b66a32d9ba00f80a06572ae4706d3f42"}
Sep 16 19:05:54.282882 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:54.282846    2562 generic.go:356] "Generic (PLEG): container finished" podID="055ee9ef-e844-43ce-970c-359b5bebb918" containerID="417fff88f833bcc6f030a4c00a362b136e7cbb590f1cd871f85d185f36c117f4" exitCode=0
Sep 16 19:05:54.283306 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:54.282930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-f8201add" event={"ID":"055ee9ef-e844-43ce-970c-359b5bebb918","Type":"ContainerDied","Data":"417fff88f833bcc6f030a4c00a362b136e7cbb590f1cd871f85d185f36c117f4"}
Sep 16 19:05:55.418016 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.417990    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:55.437375 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.437346    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-f8201add_055ee9ef-e844-43ce-970c-359b5bebb918/curl-metrics-f8201add/0.log"
Sep 16 19:05:55.459564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.459533    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-f8201add"]
Sep 16 19:05:55.465113 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.465091    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-f8201add"]
Sep 16 19:05:55.498265 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.498241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnqfh\" (UniqueName: \"kubernetes.io/projected/055ee9ef-e844-43ce-970c-359b5bebb918-kube-api-access-bnqfh\") pod \"055ee9ef-e844-43ce-970c-359b5bebb918\" (UID: \"055ee9ef-e844-43ce-970c-359b5bebb918\") "
Sep 16 19:05:55.500469 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.500448    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055ee9ef-e844-43ce-970c-359b5bebb918-kube-api-access-bnqfh" (OuterVolumeSpecName: "kube-api-access-bnqfh") pod "055ee9ef-e844-43ce-970c-359b5bebb918" (UID: "055ee9ef-e844-43ce-970c-359b5bebb918"). InnerVolumeSpecName "kube-api-access-bnqfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:55.599544 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:55.599513    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnqfh\" (UniqueName: \"kubernetes.io/projected/055ee9ef-e844-43ce-970c-359b5bebb918-kube-api-access-bnqfh\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:05:56.121914 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:56.121883    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="055ee9ef-e844-43ce-970c-359b5bebb918" path="/var/lib/kubelet/pods/055ee9ef-e844-43ce-970c-359b5bebb918/volumes"
Sep 16 19:05:56.292470 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:56.292442    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-f8201add"
Sep 16 19:05:56.292643 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:05:56.292460    2562 scope.go:117] "RemoveContainer" containerID="417fff88f833bcc6f030a4c00a362b136e7cbb590f1cd871f85d185f36c117f4"
Sep 16 19:06:00.843213 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.843180    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-43e677ae"]
Sep 16 19:06:00.843637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.843525    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="055ee9ef-e844-43ce-970c-359b5bebb918" containerName="curl-metrics-f8201add"
Sep 16 19:06:00.843637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.843537    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="055ee9ef-e844-43ce-970c-359b5bebb918" containerName="curl-metrics-f8201add"
Sep 16 19:06:00.843637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.843583    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="055ee9ef-e844-43ce-970c-359b5bebb918" containerName="curl-metrics-f8201add"
Sep 16 19:06:00.846402 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.846376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:00.848837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.848818    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6tm48\""
Sep 16 19:06:00.848929 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.848841    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 19:06:00.848929 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.848868    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 19:06:00.852364 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.852339    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-43e677ae"]
Sep 16 19:06:00.947195 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:00.947162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thz4c\" (UniqueName: \"kubernetes.io/projected/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc-kube-api-access-thz4c\") pod \"curl-metrics-43e677ae\" (UID: \"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc\") " pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:01.047631 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:01.047574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thz4c\" (UniqueName: \"kubernetes.io/projected/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc-kube-api-access-thz4c\") pod \"curl-metrics-43e677ae\" (UID: \"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc\") " pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:01.056924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:01.056893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thz4c\" (UniqueName: \"kubernetes.io/projected/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc-kube-api-access-thz4c\") pod \"curl-metrics-43e677ae\" (UID: \"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc\") " pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:01.156047 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:01.155939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:01.421542 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:01.421473    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-43e677ae"]
Sep 16 19:06:01.425048 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:06:01.425019    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3ac38cf_cfc5_4236_81a5_dbadae2b69dc.slice/crio-a4c4df773c879d6bb0d3a8cdb4e4a8f88df8d464c1fbfa324cbbd346d864453c WatchSource:0}: Error finding container a4c4df773c879d6bb0d3a8cdb4e4a8f88df8d464c1fbfa324cbbd346d864453c: Status 404 returned error can't find the container with id a4c4df773c879d6bb0d3a8cdb4e4a8f88df8d464c1fbfa324cbbd346d864453c
Sep 16 19:06:02.319617 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:02.319531    2562 generic.go:356] "Generic (PLEG): container finished" podID="f3ac38cf-cfc5-4236-81a5-dbadae2b69dc" containerID="df4346e17f2b43d8d20b8b805b3b58e732f490a1686593346e3f68e31e3d6961" exitCode=0
Sep 16 19:06:02.319951 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:02.319617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-43e677ae" event={"ID":"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc","Type":"ContainerDied","Data":"df4346e17f2b43d8d20b8b805b3b58e732f490a1686593346e3f68e31e3d6961"}
Sep 16 19:06:02.319951 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:02.319655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-43e677ae" event={"ID":"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc","Type":"ContainerStarted","Data":"a4c4df773c879d6bb0d3a8cdb4e4a8f88df8d464c1fbfa324cbbd346d864453c"}
Sep 16 19:06:03.449143 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.449113    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:03.469700 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.469665    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-43e677ae_f3ac38cf-cfc5-4236-81a5-dbadae2b69dc/curl-metrics-43e677ae/0.log"
Sep 16 19:06:03.492141 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.492117    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-43e677ae"]
Sep 16 19:06:03.496016 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.495993    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-43e677ae"]
Sep 16 19:06:03.568858 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.568831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thz4c\" (UniqueName: \"kubernetes.io/projected/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc-kube-api-access-thz4c\") pod \"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc\" (UID: \"f3ac38cf-cfc5-4236-81a5-dbadae2b69dc\") "
Sep 16 19:06:03.571019 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.570995    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc-kube-api-access-thz4c" (OuterVolumeSpecName: "kube-api-access-thz4c") pod "f3ac38cf-cfc5-4236-81a5-dbadae2b69dc" (UID: "f3ac38cf-cfc5-4236-81a5-dbadae2b69dc"). InnerVolumeSpecName "kube-api-access-thz4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:03.669680 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:03.669649    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thz4c\" (UniqueName: \"kubernetes.io/projected/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc-kube-api-access-thz4c\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:06:04.121669 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:04.121637    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ac38cf-cfc5-4236-81a5-dbadae2b69dc" path="/var/lib/kubelet/pods/f3ac38cf-cfc5-4236-81a5-dbadae2b69dc/volumes"
Sep 16 19:06:04.329151 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:04.329120    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-43e677ae"
Sep 16 19:06:04.329356 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:04.329158    2562 scope.go:117] "RemoveContainer" containerID="df4346e17f2b43d8d20b8b805b3b58e732f490a1686593346e3f68e31e3d6961"
Sep 16 19:06:08.883361 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.883302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:08.929234 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.929194    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"]
Sep 16 19:06:08.929618 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.929601    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ac38cf-cfc5-4236-81a5-dbadae2b69dc" containerName="curl-metrics-43e677ae"
Sep 16 19:06:08.929708 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.929622    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ac38cf-cfc5-4236-81a5-dbadae2b69dc" containerName="curl-metrics-43e677ae"
Sep 16 19:06:08.929708 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.929694    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ac38cf-cfc5-4236-81a5-dbadae2b69dc" containerName="curl-metrics-43e677ae"
Sep 16 19:06:08.932430 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.932409    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:08.934857 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.934833    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 19:06:08.934960 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.934873    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:06:08.935064 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.935040    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:06:08.935198 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.935113    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:06:08.935283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.935266    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:06:08.935381 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.935310    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 16 19:06:08.935381 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.935374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:06:08.943422 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:08.943401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"]
Sep 16 19:06:09.018955 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.018920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-cabundle-cert\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.018955 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.018958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.019165 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.019030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7a7853-6ebf-45db-bb24-9cf42344db95-proxy-tls\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.019165 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.019057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c7a7853-6ebf-45db-bb24-9cf42344db95-kserve-provision-location\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.019165 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.019082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt22p\" (UniqueName: \"kubernetes.io/projected/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-api-access-mt22p\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.021408 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.021370    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:09.119819 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.119781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-cabundle-cert\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.119995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.119824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.119995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.119859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7a7853-6ebf-45db-bb24-9cf42344db95-proxy-tls\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.119995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.119884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c7a7853-6ebf-45db-bb24-9cf42344db95-kserve-provision-location\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.119995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.119906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt22p\" (UniqueName: \"kubernetes.io/projected/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-api-access-mt22p\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.120366 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.120345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c7a7853-6ebf-45db-bb24-9cf42344db95-kserve-provision-location\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.120623 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.120605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.120675 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.120641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-cabundle-cert\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.122694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.122673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7a7853-6ebf-45db-bb24-9cf42344db95-proxy-tls\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.128756 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.128731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt22p\" (UniqueName: \"kubernetes.io/projected/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-api-access-mt22p\") pod \"isvc-sklearn-predictor-d96586f44-qrtkr\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.242723 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.242652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:09.513436 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:09.513413    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"]
Sep 16 19:06:09.516034 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:06:09.516005    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7a7853_6ebf_45db_bb24_9cf42344db95.slice/crio-28390469a3b0e0ab4fbe5aa556f15a00023ac1e8c024d9bd75e08fecdfb375e1 WatchSource:0}: Error finding container 28390469a3b0e0ab4fbe5aa556f15a00023ac1e8c024d9bd75e08fecdfb375e1: Status 404 returned error can't find the container with id 28390469a3b0e0ab4fbe5aa556f15a00023ac1e8c024d9bd75e08fecdfb375e1
Sep 16 19:06:10.357735 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:10.357702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerStarted","Data":"36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff"}
Sep 16 19:06:10.358121 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:10.357756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerStarted","Data":"28390469a3b0e0ab4fbe5aa556f15a00023ac1e8c024d9bd75e08fecdfb375e1"}
Sep 16 19:06:14.375555 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:14.375519    2562 generic.go:356] "Generic (PLEG): container finished" podID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerID="36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff" exitCode=0
Sep 16 19:06:14.375945 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:14.375579    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerDied","Data":"36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff"}
Sep 16 19:06:15.381957 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:15.381921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerStarted","Data":"61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4"}
Sep 16 19:06:15.382452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:15.381991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerStarted","Data":"cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9"}
Sep 16 19:06:15.382452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:15.382210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:15.408375 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:15.408310    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podStartSLOduration=3.537705017 podStartE2EDuration="7.408296664s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.870591647s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:15.405916409 +0000 UTC m=+1377.822619168" watchObservedRunningTime="2026-09-16 19:06:15.408296664 +0000 UTC m=+1377.824999423"
Sep 16 19:06:16.386572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:16.386539    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:16.387640 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:16.387616    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:06:17.392529 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:17.392494    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:06:22.398442 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:22.398413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:06:22.399174 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:22.399137    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:06:32.399414 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:32.399373    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:06:42.399477 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:42.399433    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:06:52.399334 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:06:52.399282    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:07:02.399850 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:02.399810    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:07:07.507641 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:07.507594    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:12.399239 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:12.399154    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:07:12.653019 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:12.652930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:12.981796 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:12.981697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:18.276511 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:18.276448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:22.399276 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:22.399230    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:07:32.400052 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:32.400022    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:07:39.015001 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:39.014966    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"]
Sep 16 19:07:39.015607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:39.015249    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" containerID="cri-o://cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9" gracePeriod=30
Sep 16 19:07:39.015607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:39.015355    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kube-rbac-proxy" containerID="cri-o://61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4" gracePeriod=30
Sep 16 19:07:39.041962 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:39.041933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:39.736086 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:39.736054    2562 generic.go:356] "Generic (PLEG): container finished" podID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerID="61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4" exitCode=2
Sep 16 19:07:39.736264 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:39.736127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerDied","Data":"61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4"}
Sep 16 19:07:42.393475 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:42.393432    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 16 19:07:42.400010 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:42.399952    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 16 19:07:43.551293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.551270    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:07:43.632867 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.632840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-rbac-proxy-sar-config\") pod \"7c7a7853-6ebf-45db-bb24-9cf42344db95\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") "
Sep 16 19:07:43.633038 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.632883    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-cabundle-cert\") pod \"7c7a7853-6ebf-45db-bb24-9cf42344db95\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") "
Sep 16 19:07:43.633038 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.632912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt22p\" (UniqueName: \"kubernetes.io/projected/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-api-access-mt22p\") pod \"7c7a7853-6ebf-45db-bb24-9cf42344db95\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") "
Sep 16 19:07:43.633038 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.632947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c7a7853-6ebf-45db-bb24-9cf42344db95-kserve-provision-location\") pod \"7c7a7853-6ebf-45db-bb24-9cf42344db95\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") "
Sep 16 19:07:43.633038 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.632967    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7a7853-6ebf-45db-bb24-9cf42344db95-proxy-tls\") pod \"7c7a7853-6ebf-45db-bb24-9cf42344db95\" (UID: \"7c7a7853-6ebf-45db-bb24-9cf42344db95\") "
Sep 16 19:07:43.633237 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.633198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c7a7853-6ebf-45db-bb24-9cf42344db95" (UID: "7c7a7853-6ebf-45db-bb24-9cf42344db95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:43.633296 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.633263    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7a7853-6ebf-45db-bb24-9cf42344db95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c7a7853-6ebf-45db-bb24-9cf42344db95" (UID: "7c7a7853-6ebf-45db-bb24-9cf42344db95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:43.633296 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.633260    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c7a7853-6ebf-45db-bb24-9cf42344db95" (UID: "7c7a7853-6ebf-45db-bb24-9cf42344db95"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:43.635223 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.635195    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-api-access-mt22p" (OuterVolumeSpecName: "kube-api-access-mt22p") pod "7c7a7853-6ebf-45db-bb24-9cf42344db95" (UID: "7c7a7853-6ebf-45db-bb24-9cf42344db95"). InnerVolumeSpecName "kube-api-access-mt22p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:43.635223 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.635212    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7a7853-6ebf-45db-bb24-9cf42344db95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c7a7853-6ebf-45db-bb24-9cf42344db95" (UID: "7c7a7853-6ebf-45db-bb24-9cf42344db95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:43.733810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.733712    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.733810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.733756    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c7a7853-6ebf-45db-bb24-9cf42344db95-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.733810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.733770    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt22p\" (UniqueName: \"kubernetes.io/projected/7c7a7853-6ebf-45db-bb24-9cf42344db95-kube-api-access-mt22p\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.733810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.733783    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c7a7853-6ebf-45db-bb24-9cf42344db95-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.733810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.733797    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c7a7853-6ebf-45db-bb24-9cf42344db95-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:07:43.753180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.753145    2562 generic.go:356] "Generic (PLEG): container finished" podID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerID="cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9" exitCode=0
Sep 16 19:07:43.753302 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.753202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerDied","Data":"cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9"}
Sep 16 19:07:43.753302 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.753215    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"
Sep 16 19:07:43.753302 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.753249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr" event={"ID":"7c7a7853-6ebf-45db-bb24-9cf42344db95","Type":"ContainerDied","Data":"28390469a3b0e0ab4fbe5aa556f15a00023ac1e8c024d9bd75e08fecdfb375e1"}
Sep 16 19:07:43.753302 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.753274    2562 scope.go:117] "RemoveContainer" containerID="61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4"
Sep 16 19:07:43.767872 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.767828    2562 scope.go:117] "RemoveContainer" containerID="cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9"
Sep 16 19:07:43.775714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.775698    2562 scope.go:117] "RemoveContainer" containerID="36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff"
Sep 16 19:07:43.778551 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.778529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"]
Sep 16 19:07:43.781807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.781785    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-d96586f44-qrtkr"]
Sep 16 19:07:43.784305 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.784290    2562 scope.go:117] "RemoveContainer" containerID="61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4"
Sep 16 19:07:43.784648 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:07:43.784628    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4\": container with ID starting with 61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4 not found: ID does not exist" containerID="61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4"
Sep 16 19:07:43.784696 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.784660    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4"} err="failed to get container status \"61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4\": rpc error: code = NotFound desc = could not find container \"61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4\": container with ID starting with 61a931d3e3c9155784445d0c978688478001f2ccd129f9a086b84500c8e7a6d4 not found: ID does not exist"
Sep 16 19:07:43.784696 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.784678    2562 scope.go:117] "RemoveContainer" containerID="cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9"
Sep 16 19:07:43.784943 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:07:43.784925    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9\": container with ID starting with cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9 not found: ID does not exist" containerID="cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9"
Sep 16 19:07:43.784999 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.784948    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9"} err="failed to get container status \"cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9\": rpc error: code = NotFound desc = could not find container \"cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9\": container with ID starting with cb34465b958e3c1f219eb97d76f9ebc509874af9eecc2ba0ddc8b5d2da5b3ea9 not found: ID does not exist"
Sep 16 19:07:43.784999 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.784963    2562 scope.go:117] "RemoveContainer" containerID="36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff"
Sep 16 19:07:43.785189 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:07:43.785173    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff\": container with ID starting with 36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff not found: ID does not exist" containerID="36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff"
Sep 16 19:07:43.785232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:43.785196    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff"} err="failed to get container status \"36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff\": rpc error: code = NotFound desc = could not find container \"36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff\": container with ID starting with 36d75e028e4e358cfde559d592a6234e93fe37d3d6b224c46933fb8ea3f1a6ff not found: ID does not exist"
Sep 16 19:07:44.123479 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.123450    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" path="/var/lib/kubelet/pods/7c7a7853-6ebf-45db-bb24-9cf42344db95/volumes"
Sep 16 19:07:44.143354 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.143292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:44.177216 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177190    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"]
Sep 16 19:07:44.177515 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177503    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kube-rbac-proxy"
Sep 16 19:07:44.177569 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177518    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kube-rbac-proxy"
Sep 16 19:07:44.177569 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177537    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container"
Sep 16 19:07:44.177569 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177542    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container"
Sep 16 19:07:44.177569 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177552    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="storage-initializer"
Sep 16 19:07:44.177569 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177558    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="storage-initializer"
Sep 16 19:07:44.177765 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177609    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kserve-container"
Sep 16 19:07:44.177765 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.177617    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7a7853-6ebf-45db-bb24-9cf42344db95" containerName="kube-rbac-proxy"
Sep 16 19:07:44.182122 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.182106    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.184598 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.184572    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:07:44.184727 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.184654    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:07:44.184727 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.184674    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:07:44.184727 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.184665    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:07:44.184875 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.184654    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:07:44.184939 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.184922    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 16 19:07:44.185064 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.185047    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:07:44.189953 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.189926    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"]
Sep 16 19:07:44.237525 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.237500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.237637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.237551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecda4d8a-db9d-481b-b418-8ddced75ba08-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.237637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.237578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.237637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.237600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2nzv\" (UniqueName: \"kubernetes.io/projected/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-api-access-h2nzv\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.237637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.237631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.338658 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.338629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecda4d8a-db9d-481b-b418-8ddced75ba08-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.338807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.338665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.338807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.338684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2nzv\" (UniqueName: \"kubernetes.io/projected/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-api-access-h2nzv\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.338807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.338703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.338807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.338762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.339062 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:07:44.338815    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:07:44.339062 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:07:44.338896    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls podName:ecda4d8a-db9d-481b-b418-8ddced75ba08 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:44.838874299 +0000 UTC m=+1467.255577053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" (UID: "ecda4d8a-db9d-481b-b418-8ddced75ba08") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:07:44.339062 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.339011    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecda4d8a-db9d-481b-b418-8ddced75ba08-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.339377 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.339361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.339441 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.339426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.347296 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.347274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2nzv\" (UniqueName: \"kubernetes.io/projected/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-api-access-h2nzv\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.531918 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.531832    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:44.842515 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.842477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:44.845187 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:44.845156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:45.092498 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:45.092462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:07:45.366470 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:45.366256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"]
Sep 16 19:07:45.369126 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:07:45.369096    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecda4d8a_db9d_481b_b418_8ddced75ba08.slice/crio-09de6595381980a794250c58608e9e66730f00774e5fecbc021d04bb6125194e WatchSource:0}: Error finding container 09de6595381980a794250c58608e9e66730f00774e5fecbc021d04bb6125194e: Status 404 returned error can't find the container with id 09de6595381980a794250c58608e9e66730f00774e5fecbc021d04bb6125194e
Sep 16 19:07:45.764107 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:45.764030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerStarted","Data":"d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23"}
Sep 16 19:07:45.764107 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:45.764085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerStarted","Data":"09de6595381980a794250c58608e9e66730f00774e5fecbc021d04bb6125194e"}
Sep 16 19:07:49.787358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:49.787301    2562 generic.go:356] "Generic (PLEG): container finished" podID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerID="d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23" exitCode=0
Sep 16 19:07:49.787358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:07:49.787358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerDied","Data":"d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23"}
Sep 16 19:09:30.789571 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:30.789529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:34.463424 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:34.463301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:34.640991 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:34.640955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.404116 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:55.404077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerStarted","Data":"fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165"}
Sep 16 19:09:55.404580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:55.404152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerStarted","Data":"93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23"}
Sep 16 19:09:55.404580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:55.404187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:09:55.404580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:55.404222    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:09:55.425845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:09:55.425801    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podStartSLOduration=3.33482182 podStartE2EDuration="2m11.425786312s" totalImagesPullingTime="2m4.528092906s" totalInitContainerRuntime="3.562871586s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:55.423620926 +0000 UTC m=+1597.840323686" watchObservedRunningTime="2026-09-16 19:09:55.425786312 +0000 UTC m=+1597.842489070"
Sep 16 19:10:01.414540 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:01.414512    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:10:31.496701 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:31.496612    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:10:41.417603 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:41.417575    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:10:44.317710 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:44.317679    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"]
Sep 16 19:10:44.318109 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:44.317883    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kserve-container" containerID="cri-o://93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23" gracePeriod=30
Sep 16 19:10:44.318109 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:44.317932    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kube-rbac-proxy" containerID="cri-o://fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165" gracePeriod=30
Sep 16 19:10:44.350305 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:44.350267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:44.612157 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:44.612069    2562 generic.go:356] "Generic (PLEG): container finished" podID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerID="fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165" exitCode=2
Sep 16 19:10:44.612157 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:44.612138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerDied","Data":"fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165"}
Sep 16 19:10:46.409743 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:46.409691    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 16 19:10:49.465051 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.465010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:49.498125 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.498087    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"]
Sep 16 19:10:49.500714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.500695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.502919 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.502899    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 16 19:10:49.503037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.502949    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:10:49.513480 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.513461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"]
Sep 16 19:10:49.531014 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.530985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.531182 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.531051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.531182 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.531120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrs6s\" (UniqueName: \"kubernetes.io/projected/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-api-access-qrs6s\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.531295 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.531183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.531295 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.531237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.632385 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.632348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.632580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.632396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.632580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.632432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.632580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.632472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrs6s\" (UniqueName: \"kubernetes.io/projected/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-api-access-qrs6s\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.632580 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.632502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.632812 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:10:49.632598    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 16 19:10:49.632812 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:10:49.632672    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls podName:c3b962f6-74b9-4b86-8ff2-424e87503f5c nodeName:}" failed. No retries permitted until 2026-09-16 19:10:50.132652998 +0000 UTC m=+1652.549355741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls") pod "isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" (UID: "c3b962f6-74b9-4b86-8ff2-424e87503f5c") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 16 19:10:49.632931 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.632913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.633142 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.633121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.633219 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.633129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:49.644097 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:49.644068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrs6s\" (UniqueName: \"kubernetes.io/projected/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-api-access-qrs6s\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:50.111110 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:50.111076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:50.138307 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:50.138278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:50.140821 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:50.140793    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:50.410708 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:50.410611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:10:50.689077 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:50.689052    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"]
Sep 16 19:10:50.691199 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:10:50.691169    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b962f6_74b9_4b86_8ff2_424e87503f5c.slice/crio-5f137759af6276eb28b667514590b20b4eaeefb5f80047656a655ef93c182495 WatchSource:0}: Error finding container 5f137759af6276eb28b667514590b20b4eaeefb5f80047656a655ef93c182495: Status 404 returned error can't find the container with id 5f137759af6276eb28b667514590b20b4eaeefb5f80047656a655ef93c182495
Sep 16 19:10:51.409111 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.409074    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 16 19:10:51.415639 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.415608    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.40:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 16 19:10:51.561009 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.560985    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:10:51.643105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.643062    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerStarted","Data":"4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0"}
Sep 16 19:10:51.643281 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.643124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerStarted","Data":"5f137759af6276eb28b667514590b20b4eaeefb5f80047656a655ef93c182495"}
Sep 16 19:10:51.645162 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.645136    2562 generic.go:356] "Generic (PLEG): container finished" podID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerID="93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23" exitCode=0
Sep 16 19:10:51.645276 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.645180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerDied","Data":"93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23"}
Sep 16 19:10:51.645276 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.645185    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"
Sep 16 19:10:51.645276 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.645223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz" event={"ID":"ecda4d8a-db9d-481b-b418-8ddced75ba08","Type":"ContainerDied","Data":"09de6595381980a794250c58608e9e66730f00774e5fecbc021d04bb6125194e"}
Sep 16 19:10:51.645276 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.645257    2562 scope.go:117] "RemoveContainer" containerID="fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165"
Sep 16 19:10:51.649987 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.649931    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-rbac-proxy-sar-config\") pod \"ecda4d8a-db9d-481b-b418-8ddced75ba08\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") "
Sep 16 19:10:51.650084 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.649988    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2nzv\" (UniqueName: \"kubernetes.io/projected/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-api-access-h2nzv\") pod \"ecda4d8a-db9d-481b-b418-8ddced75ba08\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") "
Sep 16 19:10:51.650084 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.650053    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls\") pod \"ecda4d8a-db9d-481b-b418-8ddced75ba08\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") "
Sep 16 19:10:51.650219 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.650123    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecda4d8a-db9d-481b-b418-8ddced75ba08-kserve-provision-location\") pod \"ecda4d8a-db9d-481b-b418-8ddced75ba08\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") "
Sep 16 19:10:51.650219 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.650163    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-cabundle-cert\") pod \"ecda4d8a-db9d-481b-b418-8ddced75ba08\" (UID: \"ecda4d8a-db9d-481b-b418-8ddced75ba08\") "
Sep 16 19:10:51.650460 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.650334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecda4d8a-db9d-481b-b418-8ddced75ba08" (UID: "ecda4d8a-db9d-481b-b418-8ddced75ba08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:51.650544 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.650467    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecda4d8a-db9d-481b-b418-8ddced75ba08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecda4d8a-db9d-481b-b418-8ddced75ba08" (UID: "ecda4d8a-db9d-481b-b418-8ddced75ba08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:51.650725 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.650705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecda4d8a-db9d-481b-b418-8ddced75ba08" (UID: "ecda4d8a-db9d-481b-b418-8ddced75ba08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:51.652162 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.652144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-api-access-h2nzv" (OuterVolumeSpecName: "kube-api-access-h2nzv") pod "ecda4d8a-db9d-481b-b418-8ddced75ba08" (UID: "ecda4d8a-db9d-481b-b418-8ddced75ba08"). InnerVolumeSpecName "kube-api-access-h2nzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:51.652480 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.652464    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecda4d8a-db9d-481b-b418-8ddced75ba08" (UID: "ecda4d8a-db9d-481b-b418-8ddced75ba08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:51.653669 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.653655    2562 scope.go:117] "RemoveContainer" containerID="93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23"
Sep 16 19:10:51.661237 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.661216    2562 scope.go:117] "RemoveContainer" containerID="d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23"
Sep 16 19:10:51.669683 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.669641    2562 scope.go:117] "RemoveContainer" containerID="fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165"
Sep 16 19:10:51.669947 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:10:51.669927    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165\": container with ID starting with fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165 not found: ID does not exist" containerID="fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165"
Sep 16 19:10:51.670011 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.669952    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165"} err="failed to get container status \"fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165\": rpc error: code = NotFound desc = could not find container \"fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165\": container with ID starting with fd0db59f539b709b33d62a23b72e3de8ee0a976ae98c344e9df98006a57c1165 not found: ID does not exist"
Sep 16 19:10:51.670011 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.669969    2562 scope.go:117] "RemoveContainer" containerID="93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23"
Sep 16 19:10:51.670186 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:10:51.670172    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23\": container with ID starting with 93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23 not found: ID does not exist" containerID="93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23"
Sep 16 19:10:51.670235 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.670189    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23"} err="failed to get container status \"93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23\": rpc error: code = NotFound desc = could not find container \"93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23\": container with ID starting with 93d13ebaf62e19a066a267289a26b0f8af0b67fb465837e4a8bf9cb6768aac23 not found: ID does not exist"
Sep 16 19:10:51.670235 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.670212    2562 scope.go:117] "RemoveContainer" containerID="d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23"
Sep 16 19:10:51.670476 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:10:51.670461    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23\": container with ID starting with d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23 not found: ID does not exist" containerID="d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23"
Sep 16 19:10:51.670529 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.670479    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23"} err="failed to get container status \"d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23\": rpc error: code = NotFound desc = could not find container \"d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23\": container with ID starting with d4527478e660a883cc8c4026c56e8d8ab69019d0e1ba202ff27e951034760a23 not found: ID does not exist"
Sep 16 19:10:51.751201 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.751166    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecda4d8a-db9d-481b-b418-8ddced75ba08-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:10:51.751201 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.751197    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecda4d8a-db9d-481b-b418-8ddced75ba08-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:10:51.751201 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.751209    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:10:51.751740 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.751220    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:10:51.751740 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.751232    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2nzv\" (UniqueName: \"kubernetes.io/projected/ecda4d8a-db9d-481b-b418-8ddced75ba08-kube-api-access-h2nzv\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:10:51.967997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.967963    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"]
Sep 16 19:10:51.974335 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:51.974296    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-84ptz"]
Sep 16 19:10:52.123028 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:10:52.122994    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" path="/var/lib/kubelet/pods/ecda4d8a-db9d-481b-b418-8ddced75ba08/volumes"
Sep 16 19:11:08.724406 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:08.724369    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerID="4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0" exitCode=0
Sep 16 19:11:08.724858 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:08.724440    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerDied","Data":"4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0"}
Sep 16 19:11:09.731176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:09.731138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerStarted","Data":"9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05"}
Sep 16 19:11:09.731684 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:09.731200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerStarted","Data":"3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc"}
Sep 16 19:11:09.731684 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:09.731464    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:11:09.731684 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:09.731595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:11:09.732843 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:09.732813    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:11:09.753106 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:09.753056    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" podStartSLOduration=2.933473467 podStartE2EDuration="20.753044742s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.819571275s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:09.751628314 +0000 UTC m=+1672.168331072" watchObservedRunningTime="2026-09-16 19:11:09.753044742 +0000 UTC m=+1672.169747568"
Sep 16 19:11:10.736117 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:10.736080    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:11:15.741258 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:15.741223    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:11:15.741874 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:15.741844    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 16 19:11:17.519557 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:17.519523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:20.750088 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:20.750055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:25.743122 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:25.743093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:11:25.880646 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:25.880597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:26.263120 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:26.263087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:32.280196 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:32.280111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"]
Sep 16 19:11:32.280623 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:32.280364    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container" containerID="cri-o://3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc" gracePeriod=30
Sep 16 19:11:32.280623 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:32.280410    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kube-rbac-proxy" containerID="cri-o://9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05" gracePeriod=30
Sep 16 19:11:32.307801 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:32.307763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:32.848894 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:32.848855    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerID="9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05" exitCode=2
Sep 16 19:11:32.849067 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:32.848910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerDied","Data":"9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05"}
Sep 16 19:11:33.125358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.125313    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:11:33.213371 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213337    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls\") pod \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") "
Sep 16 19:11:33.213530 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213406    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kserve-provision-location\") pod \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") "
Sep 16 19:11:33.213530 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-rbac-proxy-sar-config\") pod \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") "
Sep 16 19:11:33.213530 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213453    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrs6s\" (UniqueName: \"kubernetes.io/projected/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-api-access-qrs6s\") pod \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") "
Sep 16 19:11:33.213530 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213475    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-cabundle-cert\") pod \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\" (UID: \"c3b962f6-74b9-4b86-8ff2-424e87503f5c\") "
Sep 16 19:11:33.213845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213822    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3b962f6-74b9-4b86-8ff2-424e87503f5c" (UID: "c3b962f6-74b9-4b86-8ff2-424e87503f5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:33.213919 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.213850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3b962f6-74b9-4b86-8ff2-424e87503f5c" (UID: "c3b962f6-74b9-4b86-8ff2-424e87503f5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:33.215734 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.215713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-api-access-qrs6s" (OuterVolumeSpecName: "kube-api-access-qrs6s") pod "c3b962f6-74b9-4b86-8ff2-424e87503f5c" (UID: "c3b962f6-74b9-4b86-8ff2-424e87503f5c"). InnerVolumeSpecName "kube-api-access-qrs6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:33.215811 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.215713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3b962f6-74b9-4b86-8ff2-424e87503f5c" (UID: "c3b962f6-74b9-4b86-8ff2-424e87503f5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:33.240816 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.240781    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3b962f6-74b9-4b86-8ff2-424e87503f5c" (UID: "c3b962f6-74b9-4b86-8ff2-424e87503f5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.314946 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.314915    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3b962f6-74b9-4b86-8ff2-424e87503f5c-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.314946 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.314943    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.314946 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.314953    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.315358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.314962    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrs6s\" (UniqueName: \"kubernetes.io/projected/c3b962f6-74b9-4b86-8ff2-424e87503f5c-kube-api-access-qrs6s\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.315358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.314971    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3b962f6-74b9-4b86-8ff2-424e87503f5c-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.855976 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.855935    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerID="3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc" exitCode=0
Sep 16 19:11:33.856153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.856015    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"
Sep 16 19:11:33.856153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.856025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerDied","Data":"3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc"}
Sep 16 19:11:33.856153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.856090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv" event={"ID":"c3b962f6-74b9-4b86-8ff2-424e87503f5c","Type":"ContainerDied","Data":"5f137759af6276eb28b667514590b20b4eaeefb5f80047656a655ef93c182495"}
Sep 16 19:11:33.856153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.856116    2562 scope.go:117] "RemoveContainer" containerID="9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05"
Sep 16 19:11:33.865247 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.865232    2562 scope.go:117] "RemoveContainer" containerID="3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc"
Sep 16 19:11:33.873159 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.873138    2562 scope.go:117] "RemoveContainer" containerID="4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0"
Sep 16 19:11:33.876293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.876267    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"]
Sep 16 19:11:33.881629 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.881607    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-9f86f76dd-7m9nv"]
Sep 16 19:11:33.881786 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.881769    2562 scope.go:117] "RemoveContainer" containerID="9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05"
Sep 16 19:11:33.882071 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:11:33.882045    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05\": container with ID starting with 9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05 not found: ID does not exist" containerID="9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05"
Sep 16 19:11:33.882168 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.882075    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05"} err="failed to get container status \"9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05\": rpc error: code = NotFound desc = could not find container \"9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05\": container with ID starting with 9b390410b0603cb1c69dc1eb77d9b2482a25d174213d3875c7f04f8c5701fd05 not found: ID does not exist"
Sep 16 19:11:33.882168 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.882094    2562 scope.go:117] "RemoveContainer" containerID="3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc"
Sep 16 19:11:33.882424 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:11:33.882397    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc\": container with ID starting with 3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc not found: ID does not exist" containerID="3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc"
Sep 16 19:11:33.882487 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.882433    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc"} err="failed to get container status \"3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc\": rpc error: code = NotFound desc = could not find container \"3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc\": container with ID starting with 3a26f8fd6173987fc4c00b0c1a8f3df9110aee2598cbf690535795aa8063d4bc not found: ID does not exist"
Sep 16 19:11:33.882487 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.882453    2562 scope.go:117] "RemoveContainer" containerID="4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0"
Sep 16 19:11:33.882703 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:11:33.882688    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0\": container with ID starting with 4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0 not found: ID does not exist" containerID="4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0"
Sep 16 19:11:33.882753 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:33.882707    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0"} err="failed to get container status \"4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0\": rpc error: code = NotFound desc = could not find container \"4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0\": container with ID starting with 4b10efb194e7b703db25bdbe3d85b47b3b4667797ae6c7f93010c2942749e1b0 not found: ID does not exist"
Sep 16 19:11:34.122739 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:34.122665    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" path="/var/lib/kubelet/pods/c3b962f6-74b9-4b86-8ff2-424e87503f5c/volumes"
Sep 16 19:11:37.422199 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.422162    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:37.453463 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453432    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"]
Sep 16 19:11:37.453796 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453780    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container"
Sep 16 19:11:37.453871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453799    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container"
Sep 16 19:11:37.453871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453828    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kserve-container"
Sep 16 19:11:37.453871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453839    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kserve-container"
Sep 16 19:11:37.453871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453850    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kube-rbac-proxy"
Sep 16 19:11:37.453871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453860    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kube-rbac-proxy"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453886    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="storage-initializer"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453895    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="storage-initializer"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453909    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="storage-initializer"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453918    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="storage-initializer"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453927    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kube-rbac-proxy"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.453936    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kube-rbac-proxy"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.454015    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kserve-container"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.454026    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kserve-container"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.454036    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b962f6-74b9-4b86-8ff2-424e87503f5c" containerName="kube-rbac-proxy"
Sep 16 19:11:37.454140 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.454045    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecda4d8a-db9d-481b-b418-8ddced75ba08" containerName="kube-rbac-proxy"
Sep 16 19:11:37.457272 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.457253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.459618 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.459598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 16 19:11:37.459728 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.459614    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:11:37.459728 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.459603    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:11:37.459835 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.459770    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:11:37.459958 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.459943    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:11:37.460495 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.460478    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:11:37.460556 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.460515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:11:37.468592 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.468574    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"]
Sep 16 19:11:37.551639 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.551606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.551802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.551666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.551802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.551705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.551802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.551736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.551926 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.551810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwsr\" (UniqueName: \"kubernetes.io/projected/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-api-access-cjwsr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.652646 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.652612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.652814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.652673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.652814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.652718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.652814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.652750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.652814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.652796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwsr\" (UniqueName: \"kubernetes.io/projected/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-api-access-cjwsr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.653226 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.653200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.653376 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.653357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.653600 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.653386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.655378 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.655356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.661096 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.661072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwsr\" (UniqueName: \"kubernetes.io/projected/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-api-access-cjwsr\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.767784 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.767713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:37.975790 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:37.975749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:38.060646 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:38.060619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"]
Sep 16 19:11:38.063680 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:11:38.063653    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19bb8d9_40f5_42a7_b1bb_815b5cc25c24.slice/crio-882b0627a09d02c67bbcc3ef68cd3e83ddc79b7df5ec26b91b34fadabf303657 WatchSource:0}: Error finding container 882b0627a09d02c67bbcc3ef68cd3e83ddc79b7df5ec26b91b34fadabf303657: Status 404 returned error can't find the container with id 882b0627a09d02c67bbcc3ef68cd3e83ddc79b7df5ec26b91b34fadabf303657
Sep 16 19:11:38.879566 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:38.879531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerStarted","Data":"7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4"}
Sep 16 19:11:38.879952 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:38.879588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerStarted","Data":"882b0627a09d02c67bbcc3ef68cd3e83ddc79b7df5ec26b91b34fadabf303657"}
Sep 16 19:11:42.897574 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:42.897541    2562 generic.go:356] "Generic (PLEG): container finished" podID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerID="7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4" exitCode=0
Sep 16 19:11:42.897973 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:42.897612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerDied","Data":"7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4"}
Sep 16 19:11:43.905479 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:43.905446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerStarted","Data":"ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af"}
Sep 16 19:11:43.905845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:43.905505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerStarted","Data":"7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda"}
Sep 16 19:11:43.905845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:43.905705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:43.905845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:43.905757    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:11:43.926915 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:43.926869    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" podStartSLOduration=3.004009911 podStartE2EDuration="6.926855283s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.922845372s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:43.925229025 +0000 UTC m=+1706.341931789" watchObservedRunningTime="2026-09-16 19:11:43.926855283 +0000 UTC m=+1706.343558042"
Sep 16 19:11:49.914701 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:11:49.914671    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:12:08.172130 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:08.172100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:13.310919 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:13.310891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:13.389471 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:13.389433    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:19.996714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:19.996671    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:12:29.917284 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:29.917254    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:12:37.537581 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:37.537549    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"]
Sep 16 19:12:37.537957 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:37.537831    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kserve-container" containerID="cri-o://7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda" gracePeriod=30
Sep 16 19:12:37.537957 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:37.537852    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kube-rbac-proxy" containerID="cri-o://ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af" gracePeriod=30
Sep 16 19:12:37.554471 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:37.554442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:38.136876 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:38.136845    2562 generic.go:356] "Generic (PLEG): container finished" podID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerID="ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af" exitCode=2
Sep 16 19:12:38.137054 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:38.136886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerDied","Data":"ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af"}
Sep 16 19:12:39.910153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:39.910107    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 16 19:12:42.668605 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.668571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:42.710553 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.710515    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"]
Sep 16 19:12:42.714134 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.714104    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.716642 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.716619    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:12:42.716747 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.716681    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 16 19:12:42.725744 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.725726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"]
Sep 16 19:12:42.801774 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.801744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.801924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.801786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0ca06a-972b-44d0-8e4a-edc8e284580b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.801924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.801855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w7g\" (UniqueName: \"kubernetes.io/projected/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-api-access-r5w7g\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.801924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.801898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.802068 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.801974    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.902776 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.902741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.902950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.902800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.902950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.902823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.902950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.902850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0ca06a-972b-44d0-8e4a-edc8e284580b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.902950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.902892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w7g\" (UniqueName: \"kubernetes.io/projected/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-api-access-r5w7g\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.903223 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.903190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.903598 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.903574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.903673 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.903635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.905754 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.905735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0ca06a-972b-44d0-8e4a-edc8e284580b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.911651 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.911609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w7g\" (UniqueName: \"kubernetes.io/projected/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-api-access-r5w7g\") pod \"isvc-sklearn-v2-predictor-7b474946d-56mbs\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:42.936173 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:42.936108    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:43.023924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:43.023887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:43.320405 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:43.320374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"]
Sep 16 19:12:43.323215 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:12:43.323184    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0ca06a_972b_44d0_8e4a_edc8e284580b.slice/crio-c3ef4336f519547d6099015e0a64cee175960cc90c9410da131793e3dbc43259 WatchSource:0}: Error finding container c3ef4336f519547d6099015e0a64cee175960cc90c9410da131793e3dbc43259: Status 404 returned error can't find the container with id c3ef4336f519547d6099015e0a64cee175960cc90c9410da131793e3dbc43259
Sep 16 19:12:44.162128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:44.162084    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerStarted","Data":"cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a"}
Sep 16 19:12:44.162513 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:44.162157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerStarted","Data":"c3ef4336f519547d6099015e0a64cee175960cc90c9410da131793e3dbc43259"}
Sep 16 19:12:44.882605 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:44.882584    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:12:45.025304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025196    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjwsr\" (UniqueName: \"kubernetes.io/projected/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-api-access-cjwsr\") pod \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") "
Sep 16 19:12:45.025304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025268    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-proxy-tls\") pod \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") "
Sep 16 19:12:45.025304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kserve-provision-location\") pod \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") "
Sep 16 19:12:45.025304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-cabundle-cert\") pod \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") "
Sep 16 19:12:45.025686 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-rbac-proxy-sar-config\") pod \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\" (UID: \"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24\") "
Sep 16 19:12:45.025749 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025657    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" (UID: "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:45.025802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" (UID: "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:45.025802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.025785    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" (UID: "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:45.028053 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.028032    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-api-access-cjwsr" (OuterVolumeSpecName: "kube-api-access-cjwsr") pod "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" (UID: "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24"). InnerVolumeSpecName "kube-api-access-cjwsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:45.028053 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.028042    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" (UID: "c19bb8d9-40f5-42a7-b1bb-815b5cc25c24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:45.126496 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.126448    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.126496 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.126488    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjwsr\" (UniqueName: \"kubernetes.io/projected/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kube-api-access-cjwsr\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.126496 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.126502    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.126731 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.126515    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.126731 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.126527    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.167943 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.167911    2562 generic.go:356] "Generic (PLEG): container finished" podID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerID="7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda" exitCode=0
Sep 16 19:12:45.168256 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.167976    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"
Sep 16 19:12:45.168256 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.167989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerDied","Data":"7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda"}
Sep 16 19:12:45.168256 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.168055    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb" event={"ID":"c19bb8d9-40f5-42a7-b1bb-815b5cc25c24","Type":"ContainerDied","Data":"882b0627a09d02c67bbcc3ef68cd3e83ddc79b7df5ec26b91b34fadabf303657"}
Sep 16 19:12:45.168256 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.168093    2562 scope.go:117] "RemoveContainer" containerID="ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af"
Sep 16 19:12:45.178534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.178367    2562 scope.go:117] "RemoveContainer" containerID="7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda"
Sep 16 19:12:45.187174 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.187156    2562 scope.go:117] "RemoveContainer" containerID="7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4"
Sep 16 19:12:45.192979 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.192957    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"]
Sep 16 19:12:45.195122 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.195107    2562 scope.go:117] "RemoveContainer" containerID="ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af"
Sep 16 19:12:45.195597 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:45.195573    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af\": container with ID starting with ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af not found: ID does not exist" containerID="ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af"
Sep 16 19:12:45.195750 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.195609    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af"} err="failed to get container status \"ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af\": rpc error: code = NotFound desc = could not find container \"ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af\": container with ID starting with ca3c34ce5efd9c4341bad7b1c4ecaca8a94625ea2cb96419dbffff6813dc08af not found: ID does not exist"
Sep 16 19:12:45.195750 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.195635    2562 scope.go:117] "RemoveContainer" containerID="7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda"
Sep 16 19:12:45.196064 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:45.195969    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda\": container with ID starting with 7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda not found: ID does not exist" containerID="7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda"
Sep 16 19:12:45.196064 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.196001    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda"} err="failed to get container status \"7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda\": rpc error: code = NotFound desc = could not find container \"7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda\": container with ID starting with 7b9582e65fd04a8440f02ab01033c49b858133a11259861fbbc9b97af29b8bda not found: ID does not exist"
Sep 16 19:12:45.196064 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.196024    2562 scope.go:117] "RemoveContainer" containerID="7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4"
Sep 16 19:12:45.196358 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:45.196337    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4\": container with ID starting with 7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4 not found: ID does not exist" containerID="7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4"
Sep 16 19:12:45.196448 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.196367    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4"} err="failed to get container status \"7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4\": rpc error: code = NotFound desc = could not find container \"7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4\": container with ID starting with 7708a4f175e741f50b42b0c07eb115fb6a222d774d4a90c16350fd5ac98f34b4 not found: ID does not exist"
Sep 16 19:12:45.197438 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:45.197421    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-qxhfb"]
Sep 16 19:12:46.122937 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:46.122903    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" path="/var/lib/kubelet/pods/c19bb8d9-40f5-42a7-b1bb-815b5cc25c24/volumes"
Sep 16 19:12:47.178676 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:47.178652    2562 generic.go:356] "Generic (PLEG): container finished" podID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerID="cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a" exitCode=0
Sep 16 19:12:47.178976 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:47.178724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerDied","Data":"cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a"}
Sep 16 19:12:48.184712 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.184678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerStarted","Data":"682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e"}
Sep 16 19:12:48.185095 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.184736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerStarted","Data":"5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5"}
Sep 16 19:12:48.185095 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.185049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:48.185201 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.185186    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:48.186422 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.186394    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:48.208422 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.208376    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podStartSLOduration=2.444374178 podStartE2EDuration="6.20836418s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763990002s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:48.206601535 +0000 UTC m=+1770.623304301" watchObservedRunningTime="2026-09-16 19:12:48.20836418 +0000 UTC m=+1770.625066938"
Sep 16 19:12:48.567310 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:48.567261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:49.189643 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:49.189602    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:50.194423 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:50.194380    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:53.640359 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.640288    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:53.697083 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697045    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"]
Sep 16 19:12:53.697486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697468    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kserve-container"
Sep 16 19:12:53.697486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697488    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kserve-container"
Sep 16 19:12:53.697638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697522    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="storage-initializer"
Sep 16 19:12:53.697638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697531    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="storage-initializer"
Sep 16 19:12:53.697638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697544    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kube-rbac-proxy"
Sep 16 19:12:53.697638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697553    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kube-rbac-proxy"
Sep 16 19:12:53.697638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697626    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kube-rbac-proxy"
Sep 16 19:12:53.697638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.697640    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19bb8d9-40f5-42a7-b1bb-815b5cc25c24" containerName="kserve-container"
Sep 16 19:12:53.700754 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.700733    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.702961 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.702935    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:12:53.703125 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.703059    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:12:53.703276 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.703228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:53.708184 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.708165    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"]
Sep 16 19:12:53.711485 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.711467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.712454 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.712435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"]
Sep 16 19:12:53.713826 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.713806    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:12:53.726058 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.726039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"]
Sep 16 19:12:53.801733 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.801733 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.801948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.801948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00181532-b2d8-41bd-9e7d-40f31a10f23b-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.801948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.802069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.802069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.801998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.802069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.802044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmbp\" (UniqueName: \"kubernetes.io/projected/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-api-access-vlmbp\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.802179 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.802086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrb6\" (UniqueName: \"kubernetes.io/projected/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-api-access-pfrb6\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.802179 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.802141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.903068 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.902973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmbp\" (UniqueName: \"kubernetes.io/projected/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-api-access-vlmbp\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.903068 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrb6\" (UniqueName: \"kubernetes.io/projected/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-api-access-pfrb6\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.903285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.903285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.903285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:53.903349    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00181532-b2d8-41bd-9e7d-40f31a10f23b-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:53.903425    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls podName:00181532-b2d8-41bd-9e7d-40f31a10f23b nodeName:}" failed. No retries permitted until 2026-09-16 19:12:54.403404623 +0000 UTC m=+1776.820107361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls") pod "isvc-canary-create-predictor-6cc94c6f4f-qkb8c" (UID: "00181532-b2d8-41bd-9e7d-40f31a10f23b") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 16 19:12:53.903484 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.903850 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:53.903519    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:12:53.903850 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:12:53.903591    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls podName:9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51 nodeName:}" failed. No retries permitted until 2026-09-16 19:12:54.403572534 +0000 UTC m=+1776.820275291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls") pod "isvc-canary-create-v2-predictor-7855d67596-sgvxp" (UID: "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 16 19:12:53.903850 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.903850 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.903769    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00181532-b2d8-41bd-9e7d-40f31a10f23b-kserve-provision-location\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.904074 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.904060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.904134 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.904080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.904190 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.904173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-cabundle-cert\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.904399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.904378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.911874 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.911841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmbp\" (UniqueName: \"kubernetes.io/projected/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-api-access-vlmbp\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:53.911992 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.911844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrb6\" (UniqueName: \"kubernetes.io/projected/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-api-access-pfrb6\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:53.987515 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:53.987483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:54.090986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.090956    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:54.407589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.407551    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:54.407751 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.407627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:54.410185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.410159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7855d67596-sgvxp\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:54.410304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.410192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls\") pod \"isvc-canary-create-predictor-6cc94c6f4f-qkb8c\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:54.612129 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.612092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:12:54.621311 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.621291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:12:54.895543 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.895511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"]
Sep 16 19:12:54.898920 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:12:54.898892    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6f47ab_fcbe_42c1_b1b5_5a0d23bbbe51.slice/crio-8df22d886b35def04e8da4c9e431526500eaa5a67282d2fa476f997dab8037e4 WatchSource:0}: Error finding container 8df22d886b35def04e8da4c9e431526500eaa5a67282d2fa476f997dab8037e4: Status 404 returned error can't find the container with id 8df22d886b35def04e8da4c9e431526500eaa5a67282d2fa476f997dab8037e4
Sep 16 19:12:54.910188 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:54.910166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"]
Sep 16 19:12:54.912394 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:12:54.912367    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00181532_b2d8_41bd_9e7d_40f31a10f23b.slice/crio-a7bbab81173b2a36b50f47df09b2a6e2ad774f8690e92328d2bb5a5b24eeab6b WatchSource:0}: Error finding container a7bbab81173b2a36b50f47df09b2a6e2ad774f8690e92328d2bb5a5b24eeab6b: Status 404 returned error can't find the container with id a7bbab81173b2a36b50f47df09b2a6e2ad774f8690e92328d2bb5a5b24eeab6b
Sep 16 19:12:55.199202 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:55.199125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:12:55.199914 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:55.199889    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:12:55.218767 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:55.218733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerStarted","Data":"f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a"}
Sep 16 19:12:55.218767 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:55.218803    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerStarted","Data":"a7bbab81173b2a36b50f47df09b2a6e2ad774f8690e92328d2bb5a5b24eeab6b"}
Sep 16 19:12:55.220999 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:55.220972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerStarted","Data":"5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a"}
Sep 16 19:12:55.221128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:55.221018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerStarted","Data":"8df22d886b35def04e8da4c9e431526500eaa5a67282d2fa476f997dab8037e4"}
Sep 16 19:12:59.241831 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:59.241794    2562 generic.go:356] "Generic (PLEG): container finished" podID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerID="f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a" exitCode=0
Sep 16 19:12:59.242246 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:59.241867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerDied","Data":"f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a"}
Sep 16 19:12:59.243553 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:59.243524    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerID="5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a" exitCode=0
Sep 16 19:12:59.243638 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:12:59.243570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerDied","Data":"5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a"}
Sep 16 19:13:00.249995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.249962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerStarted","Data":"040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d"}
Sep 16 19:13:00.250486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.250038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerStarted","Data":"c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857"}
Sep 16 19:13:00.250486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.250280    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:13:00.252075 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.252054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerStarted","Data":"f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042"}
Sep 16 19:13:00.252195 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.252100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerStarted","Data":"4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f"}
Sep 16 19:13:00.252367 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.252350    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:13:00.252427 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.252405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:13:00.253769 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.253747    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:00.271798 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.271720    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podStartSLOduration=3.49861063 podStartE2EDuration="7.271705686s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773095056s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:00.270633682 +0000 UTC m=+1782.687336442" watchObservedRunningTime="2026-09-16 19:13:00.271705686 +0000 UTC m=+1782.688408444"
Sep 16 19:13:00.292590 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:00.292542    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podStartSLOduration=3.491580665 podStartE2EDuration="7.292526628s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800945963s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:00.290859317 +0000 UTC m=+1782.707562079" watchObservedRunningTime="2026-09-16 19:13:00.292526628 +0000 UTC m=+1782.709229390"
Sep 16 19:13:01.257653 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:01.257612    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:01.257653 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:01.257634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:13:01.258774 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:01.258752    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:02.262052 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:02.262012    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:05.199824 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:05.199778    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:13:06.263411 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:06.263383    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:13:06.263912 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:06.263883    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:07.266636 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:07.266605    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:13:07.267193 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:07.267165    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:15.199860 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:15.199760    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:13:16.264044 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:16.264008    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:17.267846 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:17.267801    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:25.199807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:25.199763    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:13:26.264431 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:26.264386    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:27.267204 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:27.267158    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:35.200169 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:35.200126    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:13:36.263929 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:36.263888    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:37.267698 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:37.267664    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:45.200607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:45.200555    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:13:46.264630 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:46.264594    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:47.267612 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:47.267571    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:13:55.200557 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:55.200518    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:13:56.264570 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:56.264534    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:13:57.267036 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:13:57.266997    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:05.201109 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:05.201075    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:14:06.264537 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:06.264499    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:14:07.267243 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:07.267210    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 16 19:14:12.850609 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:12.850574    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"]
Sep 16 19:14:12.851008 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:12.850850    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" containerID="cri-o://5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5" gracePeriod=30
Sep 16 19:14:12.851008 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:12.850868    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kube-rbac-proxy" containerID="cri-o://682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e" gracePeriod=30
Sep 16 19:14:12.873960 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:12.873916    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:13.583542 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:13.583506    2562 generic.go:356] "Generic (PLEG): container finished" podID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerID="682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e" exitCode=2
Sep 16 19:14:13.583717 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:13.583568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerDied","Data":"682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e"}
Sep 16 19:14:15.195281 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:15.195243    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 16 19:14:15.200633 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:15.200601    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 16 19:14:16.264474 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.264444    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:14:16.394953 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.394919    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"]
Sep 16 19:14:16.395178 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.395141    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container" containerID="cri-o://c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857" gracePeriod=30
Sep 16 19:14:16.395251 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.395206    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kube-rbac-proxy" containerID="cri-o://040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d" gracePeriod=30
Sep 16 19:14:16.396418 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.396357    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:14:16.600088 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.600049    2562 generic.go:356] "Generic (PLEG): container finished" podID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerID="040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d" exitCode=2
Sep 16 19:14:16.600262 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:16.600124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerDied","Data":"040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d"}
Sep 16 19:14:17.263256 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.263217    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 16 19:14:17.590555 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.590532    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:14:17.606607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.606580    2562 generic.go:356] "Generic (PLEG): container finished" podID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerID="5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5" exitCode=0
Sep 16 19:14:17.606738 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.606630    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"
Sep 16 19:14:17.606738 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.606665    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerDied","Data":"5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5"}
Sep 16 19:14:17.606738 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.606725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs" event={"ID":"4b0ca06a-972b-44d0-8e4a-edc8e284580b","Type":"ContainerDied","Data":"c3ef4336f519547d6099015e0a64cee175960cc90c9410da131793e3dbc43259"}
Sep 16 19:14:17.606899 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.606756    2562 scope.go:117] "RemoveContainer" containerID="682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e"
Sep 16 19:14:17.616163 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.616144    2562 scope.go:117] "RemoveContainer" containerID="5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5"
Sep 16 19:14:17.624886 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.624869    2562 scope.go:117] "RemoveContainer" containerID="cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a"
Sep 16 19:14:17.632415 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.632397    2562 scope.go:117] "RemoveContainer" containerID="682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e"
Sep 16 19:14:17.632685 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:17.632667    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e\": container with ID starting with 682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e not found: ID does not exist" containerID="682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e"
Sep 16 19:14:17.632732 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.632695    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e"} err="failed to get container status \"682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e\": rpc error: code = NotFound desc = could not find container \"682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e\": container with ID starting with 682fe41dbcfbf2798da2c1981e4cc698bb64bfc583a60c9716ae06df103dcf6e not found: ID does not exist"
Sep 16 19:14:17.632732 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.632711    2562 scope.go:117] "RemoveContainer" containerID="5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5"
Sep 16 19:14:17.632975 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:17.632959    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5\": container with ID starting with 5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5 not found: ID does not exist" containerID="5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5"
Sep 16 19:14:17.633032 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.632980    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5"} err="failed to get container status \"5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5\": rpc error: code = NotFound desc = could not find container \"5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5\": container with ID starting with 5267be0f934298dff9430493f54eb346c39ae8b414b24034fd68b5ac64f59bc5 not found: ID does not exist"
Sep 16 19:14:17.633032 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.632992    2562 scope.go:117] "RemoveContainer" containerID="cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a"
Sep 16 19:14:17.633215 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:17.633199    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a\": container with ID starting with cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a not found: ID does not exist" containerID="cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a"
Sep 16 19:14:17.633263 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.633218    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a"} err="failed to get container status \"cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a\": rpc error: code = NotFound desc = could not find container \"cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a\": container with ID starting with cee0640dddf5cd868d306176ddf5d4635632a0d7dd2ac5a93abf9d4b57d2847a not found: ID does not exist"
Sep 16 19:14:17.737564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737536    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-cabundle-cert\") pod \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") "
Sep 16 19:14:17.737675 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737580    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kserve-provision-location\") pod \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") "
Sep 16 19:14:17.737675 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737622    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0ca06a-972b-44d0-8e4a-edc8e284580b-proxy-tls\") pod \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") "
Sep 16 19:14:17.737675 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-rbac-proxy-sar-config\") pod \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") "
Sep 16 19:14:17.737808 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737706    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5w7g\" (UniqueName: \"kubernetes.io/projected/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-api-access-r5w7g\") pod \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\" (UID: \"4b0ca06a-972b-44d0-8e4a-edc8e284580b\") "
Sep 16 19:14:17.737970 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b0ca06a-972b-44d0-8e4a-edc8e284580b" (UID: "4b0ca06a-972b-44d0-8e4a-edc8e284580b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:17.737970 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.737945    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b0ca06a-972b-44d0-8e4a-edc8e284580b" (UID: "4b0ca06a-972b-44d0-8e4a-edc8e284580b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:17.738107 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.738003    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b0ca06a-972b-44d0-8e4a-edc8e284580b" (UID: "4b0ca06a-972b-44d0-8e4a-edc8e284580b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:17.739824 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.739806    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0ca06a-972b-44d0-8e4a-edc8e284580b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b0ca06a-972b-44d0-8e4a-edc8e284580b" (UID: "4b0ca06a-972b-44d0-8e4a-edc8e284580b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:17.739905 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.739884    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-api-access-r5w7g" (OuterVolumeSpecName: "kube-api-access-r5w7g") pod "4b0ca06a-972b-44d0-8e4a-edc8e284580b" (UID: "4b0ca06a-972b-44d0-8e4a-edc8e284580b"). InnerVolumeSpecName "kube-api-access-r5w7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:17.838539 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.838513    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:17.838539 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.838536    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:17.838690 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.838547    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0ca06a-972b-44d0-8e4a-edc8e284580b-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:17.838690 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.838556    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:17.838690 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.838564    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w7g\" (UniqueName: \"kubernetes.io/projected/4b0ca06a-972b-44d0-8e4a-edc8e284580b-kube-api-access-r5w7g\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:17.933187 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.933155    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"]
Sep 16 19:14:17.937620 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:17.937599    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7b474946d-56mbs"]
Sep 16 19:14:18.048763 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:18.048729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:18.122302 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:18.122220    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" path="/var/lib/kubelet/pods/4b0ca06a-972b-44d0-8e4a-edc8e284580b/volumes"
Sep 16 19:14:18.355061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:18.355022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:21.039433 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.039405    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:14:21.168822 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.168724    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00181532-b2d8-41bd-9e7d-40f31a10f23b-kserve-provision-location\") pod \"00181532-b2d8-41bd-9e7d-40f31a10f23b\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") "
Sep 16 19:14:21.168822 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.168764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls\") pod \"00181532-b2d8-41bd-9e7d-40f31a10f23b\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") "
Sep 16 19:14:21.168822 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.168786    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfrb6\" (UniqueName: \"kubernetes.io/projected/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-api-access-pfrb6\") pod \"00181532-b2d8-41bd-9e7d-40f31a10f23b\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") "
Sep 16 19:14:21.169121 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.168851    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-cabundle-cert\") pod \"00181532-b2d8-41bd-9e7d-40f31a10f23b\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") "
Sep 16 19:14:21.169121 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.168892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-rbac-proxy-sar-config\") pod \"00181532-b2d8-41bd-9e7d-40f31a10f23b\" (UID: \"00181532-b2d8-41bd-9e7d-40f31a10f23b\") "
Sep 16 19:14:21.169214 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.169126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00181532-b2d8-41bd-9e7d-40f31a10f23b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00181532-b2d8-41bd-9e7d-40f31a10f23b" (UID: "00181532-b2d8-41bd-9e7d-40f31a10f23b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:21.169267 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.169244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00181532-b2d8-41bd-9e7d-40f31a10f23b" (UID: "00181532-b2d8-41bd-9e7d-40f31a10f23b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:21.169267 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.169249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00181532-b2d8-41bd-9e7d-40f31a10f23b" (UID: "00181532-b2d8-41bd-9e7d-40f31a10f23b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:21.171013 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.170992    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00181532-b2d8-41bd-9e7d-40f31a10f23b" (UID: "00181532-b2d8-41bd-9e7d-40f31a10f23b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:21.171104 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.171085    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-api-access-pfrb6" (OuterVolumeSpecName: "kube-api-access-pfrb6") pod "00181532-b2d8-41bd-9e7d-40f31a10f23b" (UID: "00181532-b2d8-41bd-9e7d-40f31a10f23b"). InnerVolumeSpecName "kube-api-access-pfrb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:21.269726 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.269686    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:21.269726 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.269714    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00181532-b2d8-41bd-9e7d-40f31a10f23b-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:21.269726 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.269723    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00181532-b2d8-41bd-9e7d-40f31a10f23b-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:21.269726 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.269732    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfrb6\" (UniqueName: \"kubernetes.io/projected/00181532-b2d8-41bd-9e7d-40f31a10f23b-kube-api-access-pfrb6\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:21.269998 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.269740    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00181532-b2d8-41bd-9e7d-40f31a10f23b-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:21.625278 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.625241    2562 generic.go:356] "Generic (PLEG): container finished" podID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerID="c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857" exitCode=0
Sep 16 19:14:21.625452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.625334    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"
Sep 16 19:14:21.625452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.625336    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerDied","Data":"c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857"}
Sep 16 19:14:21.625452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.625398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c" event={"ID":"00181532-b2d8-41bd-9e7d-40f31a10f23b","Type":"ContainerDied","Data":"a7bbab81173b2a36b50f47df09b2a6e2ad774f8690e92328d2bb5a5b24eeab6b"}
Sep 16 19:14:21.625452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.625427    2562 scope.go:117] "RemoveContainer" containerID="040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d"
Sep 16 19:14:21.634514 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.634497    2562 scope.go:117] "RemoveContainer" containerID="c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857"
Sep 16 19:14:21.642168 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.642150    2562 scope.go:117] "RemoveContainer" containerID="f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a"
Sep 16 19:14:21.646071 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.646052    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"]
Sep 16 19:14:21.651228 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.651192    2562 scope.go:117] "RemoveContainer" containerID="040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d"
Sep 16 19:14:21.652701 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.652672    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6cc94c6f4f-qkb8c"]
Sep 16 19:14:21.655312 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:21.655288    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d\": container with ID starting with 040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d not found: ID does not exist" containerID="040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d"
Sep 16 19:14:21.655435 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.655340    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d"} err="failed to get container status \"040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d\": rpc error: code = NotFound desc = could not find container \"040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d\": container with ID starting with 040bad4ce1522243ebbfa76f0039ccb734b4e762630b8a6e8c9afc9cb0574d8d not found: ID does not exist"
Sep 16 19:14:21.655435 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.655369    2562 scope.go:117] "RemoveContainer" containerID="c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857"
Sep 16 19:14:21.655754 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:21.655735    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857\": container with ID starting with c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857 not found: ID does not exist" containerID="c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857"
Sep 16 19:14:21.655839 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.655763    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857"} err="failed to get container status \"c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857\": rpc error: code = NotFound desc = could not find container \"c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857\": container with ID starting with c154cabea100837eec7bb372b54835611bc1362f64950c9cb52441d795f91857 not found: ID does not exist"
Sep 16 19:14:21.655839 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.655784    2562 scope.go:117] "RemoveContainer" containerID="f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a"
Sep 16 19:14:21.656225 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:21.656199    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a\": container with ID starting with f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a not found: ID does not exist" containerID="f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a"
Sep 16 19:14:21.656277 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:21.656236    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a"} err="failed to get container status \"f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a\": rpc error: code = NotFound desc = could not find container \"f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a\": container with ID starting with f64e7d21eb1331917fe96ac2e7ddf62297d742f02efb604374fb81d36c7f018a not found: ID does not exist"
Sep 16 19:14:22.123011 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:22.122974    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" path="/var/lib/kubelet/pods/00181532-b2d8-41bd-9e7d-40f31a10f23b/volumes"
Sep 16 19:14:33.748996 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:33.748964    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"]
Sep 16 19:14:33.749487 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:33.749213    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" containerID="cri-o://4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f" gracePeriod=30
Sep 16 19:14:33.749487 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:33.749237    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kube-rbac-proxy" containerID="cri-o://f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042" gracePeriod=30
Sep 16 19:14:33.805631 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:33.805602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:33.899964 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:33.899917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:34.683762 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:34.683726    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerID="f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042" exitCode=2
Sep 16 19:14:34.683931 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:34.683799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerDied","Data":"f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042"}
Sep 16 19:14:36.258536 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:36.258493    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 16 19:14:36.264717 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:36.264691    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 16 19:14:38.489267 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.489245    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:14:38.517425 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517397    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kserve-provision-location\") pod \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") "
Sep 16 19:14:38.517570 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517435    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls\") pod \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") "
Sep 16 19:14:38.517570 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517512    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-rbac-proxy-sar-config\") pod \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") "
Sep 16 19:14:38.517570 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517560    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-cabundle-cert\") pod \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") "
Sep 16 19:14:38.517749 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlmbp\" (UniqueName: \"kubernetes.io/projected/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-api-access-vlmbp\") pod \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\" (UID: \"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51\") "
Sep 16 19:14:38.517830 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517745    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" (UID: "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:38.517935 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" (UID: "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:38.518046 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.517956    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" (UID: "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:38.519816 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.519785    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-api-access-vlmbp" (OuterVolumeSpecName: "kube-api-access-vlmbp") pod "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" (UID: "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51"). InnerVolumeSpecName "kube-api-access-vlmbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:38.519936 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.519850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" (UID: "9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:38.618220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.618145    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.618220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.618169    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.618220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.618178    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmbp\" (UniqueName: \"kubernetes.io/projected/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kube-api-access-vlmbp\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.618220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.618187    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.618220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.618196    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:14:38.702825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.702790    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerID="4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f" exitCode=0
Sep 16 19:14:38.702997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.702848    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"
Sep 16 19:14:38.702997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.702850    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerDied","Data":"4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f"}
Sep 16 19:14:38.702997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.702910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp" event={"ID":"9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51","Type":"ContainerDied","Data":"8df22d886b35def04e8da4c9e431526500eaa5a67282d2fa476f997dab8037e4"}
Sep 16 19:14:38.702997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.702939    2562 scope.go:117] "RemoveContainer" containerID="f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042"
Sep 16 19:14:38.712431 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.712353    2562 scope.go:117] "RemoveContainer" containerID="4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f"
Sep 16 19:14:38.720019 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.720000    2562 scope.go:117] "RemoveContainer" containerID="5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a"
Sep 16 19:14:38.725033 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.725011    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"]
Sep 16 19:14:38.727991 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.727971    2562 scope.go:117] "RemoveContainer" containerID="f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042"
Sep 16 19:14:38.728336 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:38.728295    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042\": container with ID starting with f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042 not found: ID does not exist" containerID="f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042"
Sep 16 19:14:38.728410 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.728347    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042"} err="failed to get container status \"f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042\": rpc error: code = NotFound desc = could not find container \"f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042\": container with ID starting with f6e7e4c5d8749bbb3e3d9996279c5b23cf74cd5eb52165a4db07f0fd65c9f042 not found: ID does not exist"
Sep 16 19:14:38.728410 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.728368    2562 scope.go:117] "RemoveContainer" containerID="4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f"
Sep 16 19:14:38.728651 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:38.728628    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f\": container with ID starting with 4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f not found: ID does not exist" containerID="4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f"
Sep 16 19:14:38.728694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.728660    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f"} err="failed to get container status \"4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f\": rpc error: code = NotFound desc = could not find container \"4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f\": container with ID starting with 4a45612af6f2c0d77309b8e63ffc60c7efb2acd8cc257d7e877e167fa8b4a57f not found: ID does not exist"
Sep 16 19:14:38.728694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.728682    2562 scope.go:117] "RemoveContainer" containerID="5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a"
Sep 16 19:14:38.728932 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:38.728916    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a\": container with ID starting with 5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a not found: ID does not exist" containerID="5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a"
Sep 16 19:14:38.728981 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.728939    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a"} err="failed to get container status \"5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a\": rpc error: code = NotFound desc = could not find container \"5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a\": container with ID starting with 5de5db80bfd6f275fd94552ed883212293b3bcd76871d20f9191f00586e26f4a not found: ID does not exist"
Sep 16 19:14:38.729819 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.729801    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7855d67596-sgvxp"]
Sep 16 19:14:38.892346 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.892235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:38.966924 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.966890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:38.980005 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.979975    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"]
Sep 16 19:14:38.980332 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980303    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="storage-initializer"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980332    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="storage-initializer"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980345    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980353    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980359    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980365    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980374    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="storage-initializer"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980381    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="storage-initializer"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980389    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container"
Sep 16 19:14:38.980393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980394    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980400    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980405    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980411    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980416    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980431    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="storage-initializer"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980436    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="storage-initializer"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980446    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980452    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980501    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980508    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kserve-container"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980515    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00181532-b2d8-41bd-9e7d-40f31a10f23b" containerName="kserve-container"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980520    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0ca06a-972b-44d0-8e4a-edc8e284580b" containerName="kube-rbac-proxy"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980526    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kserve-container"
Sep 16 19:14:38.980703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.980532    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" containerName="kube-rbac-proxy"
Sep 16 19:14:38.984970 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.984955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:38.988357 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.988311    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:14:38.988488 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.988449    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:14:38.988563 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.988553    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:14:38.988700 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.988683    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:38.989037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.989021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:14:38.989226 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.989209    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:14:38.990221 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.990178    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:38.999390 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:38.999372    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"]
Sep 16 19:14:39.020333 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.020289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.020464 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.020392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.020464 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.020416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43e1002a-bc08-46e9-9094-8817ad346bae-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.020550 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.020462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmldj\" (UniqueName: \"kubernetes.io/projected/43e1002a-bc08-46e9-9094-8817ad346bae-kube-api-access-qmldj\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.020550 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.020530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121198 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121198 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43e1002a-bc08-46e9-9094-8817ad346bae-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121445 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmldj\" (UniqueName: \"kubernetes.io/projected/43e1002a-bc08-46e9-9094-8817ad346bae-kube-api-access-qmldj\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121445 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121445 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121609 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:39.121460    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:14:39.121609 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:14:39.121549    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls podName:43e1002a-bc08-46e9-9094-8817ad346bae nodeName:}" failed. No retries permitted until 2026-09-16 19:14:39.621527605 +0000 UTC m=+1882.038230361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls") pod "isvc-canary-promote-predictor-b9b98bcd9-bj7dj" (UID: "43e1002a-bc08-46e9-9094-8817ad346bae") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:14:39.121718 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43e1002a-bc08-46e9-9094-8817ad346bae-kserve-provision-location\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121869 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-cabundle-cert\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.121920 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.121903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.131994 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.131975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmldj\" (UniqueName: \"kubernetes.io/projected/43e1002a-bc08-46e9-9094-8817ad346bae-kube-api-access-qmldj\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.413393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.413314    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:39.514501 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.514449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:39.624076 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.623995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.626523 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.626500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls\") pod \"isvc-canary-promote-predictor-b9b98bcd9-bj7dj\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:39.895136 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:39.895107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:40.121848 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:40.121814    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51" path="/var/lib/kubelet/pods/9b6f47ab-fcbe-42c1-b1b5-5a0d23bbbe51/volumes"
Sep 16 19:14:40.168970 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:40.168948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"]
Sep 16 19:14:40.171255 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:14:40.171222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e1002a_bc08_46e9_9094_8817ad346bae.slice/crio-08b50af1f672886ddad00b3188dc6e36eb8500e6f85820892fdfd4d99cc26249 WatchSource:0}: Error finding container 08b50af1f672886ddad00b3188dc6e36eb8500e6f85820892fdfd4d99cc26249: Status 404 returned error can't find the container with id 08b50af1f672886ddad00b3188dc6e36eb8500e6f85820892fdfd4d99cc26249
Sep 16 19:14:40.714163 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:40.714127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerStarted","Data":"7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0"}
Sep 16 19:14:40.714558 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:40.714196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerStarted","Data":"08b50af1f672886ddad00b3188dc6e36eb8500e6f85820892fdfd4d99cc26249"}
Sep 16 19:14:44.732126 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:44.732094    2562 generic.go:356] "Generic (PLEG): container finished" podID="43e1002a-bc08-46e9-9094-8817ad346bae" containerID="7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0" exitCode=0
Sep 16 19:14:44.732495 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:44.732169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerDied","Data":"7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0"}
Sep 16 19:14:45.738546 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:45.738508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerStarted","Data":"b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249"}
Sep 16 19:14:45.738921 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:45.738578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerStarted","Data":"e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b"}
Sep 16 19:14:45.738921 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:45.738789    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:45.758498 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:45.758450    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podStartSLOduration=3.947704958 podStartE2EDuration="7.75843548s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.810730522s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:45.757049613 +0000 UTC m=+1888.173752393" watchObservedRunningTime="2026-09-16 19:14:45.75843548 +0000 UTC m=+1888.175138238"
Sep 16 19:14:46.743342 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:46.743292    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:46.744575 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:46.744545    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:14:47.747624 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:47.747584    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:14:52.751832 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:52.751804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:14:52.752288 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:14:52.752230    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:02.752455 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:02.752417    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:12.753123 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:12.753077    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:22.752610 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:22.752567    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:32.753243 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:32.753196    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:42.752974 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:42.752932    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:48.255077 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:48.255040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:52.752307 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:52.752268    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 16 19:15:53.370839 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.370782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:53.411419 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.411382    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"]
Sep 16 19:15:53.414970 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.414952    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.417213 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.417190    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 16 19:15:53.417444 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.417419    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 16 19:15:53.424194 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.424171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"]
Sep 16 19:15:53.541696 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.541665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2107cc-c3ae-4c2a-b416-1f5144608083-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.541875 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.541705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsnq\" (UniqueName: \"kubernetes.io/projected/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-api-access-6xsnq\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.541875 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.541722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.541875 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.541809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.541875 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.541845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643290 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2107cc-c3ae-4c2a-b416-1f5144608083-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643290 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsnq\" (UniqueName: \"kubernetes.io/projected/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-api-access-6xsnq\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643290 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643579 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643579 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643693 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:15:53.643575    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 16 19:15:53.643693 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:15:53.643664    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls podName:5c2107cc-c3ae-4c2a-b416-1f5144608083 nodeName:}" failed. No retries permitted until 2026-09-16 19:15:54.143642113 +0000 UTC m=+1956.560344864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-ssm8l" (UID: "5c2107cc-c3ae-4c2a-b416-1f5144608083") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 16 19:15:53.643810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2107cc-c3ae-4c2a-b416-1f5144608083-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.643996 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.643978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.644071 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.644056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.652135 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.652110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsnq\" (UniqueName: \"kubernetes.io/projected/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-api-access-6xsnq\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:53.756719 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:53.756691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:54.147862 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:54.147834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:54.150304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:54.150286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-ssm8l\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:54.325981 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:54.325948    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:15:54.596368 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:54.596334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"]
Sep 16 19:15:54.599903 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:15:54.599878    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2107cc_c3ae_4c2a_b416_1f5144608083.slice/crio-496d32f7e55ebcae800b6fcae7356d0f037af1c8fe816ab9b6d55148dcc6d243 WatchSource:0}: Error finding container 496d32f7e55ebcae800b6fcae7356d0f037af1c8fe816ab9b6d55148dcc6d243: Status 404 returned error can't find the container with id 496d32f7e55ebcae800b6fcae7356d0f037af1c8fe816ab9b6d55148dcc6d243
Sep 16 19:15:55.026287 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:55.026202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerStarted","Data":"0f875f7fb157a14be4760518c978de19716b957512f0561ee3833113190e533d"}
Sep 16 19:15:55.026287 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:55.026269    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerStarted","Data":"496d32f7e55ebcae800b6fcae7356d0f037af1c8fe816ab9b6d55148dcc6d243"}
Sep 16 19:15:59.044367 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:59.044333    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerID="0f875f7fb157a14be4760518c978de19716b957512f0561ee3833113190e533d" exitCode=0
Sep 16 19:15:59.044367 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:59.044358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerDied","Data":"0f875f7fb157a14be4760518c978de19716b957512f0561ee3833113190e533d"}
Sep 16 19:15:59.045648 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:15:59.045629    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:16:01.938871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:01.938832    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"]
Sep 16 19:16:01.939576 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:01.939102    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container" containerID="cri-o://e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b" gracePeriod=30
Sep 16 19:16:01.939576 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:01.939273    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kube-rbac-proxy" containerID="cri-o://b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249" gracePeriod=30
Sep 16 19:16:01.940169 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:01.940147    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:16:02.748209 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:02.748174    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 16 19:16:03.065870 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.065839    2562 generic.go:356] "Generic (PLEG): container finished" podID="43e1002a-bc08-46e9-9094-8817ad346bae" containerID="b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249" exitCode=2
Sep 16 19:16:03.066426 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.065919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerDied","Data":"b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249"}
Sep 16 19:16:03.068121 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.068101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerStarted","Data":"1e5834cd6101beaa918ef1c5ab8fdedcb71b003bc27ef0e3b1bb88d1815ae3ff"}
Sep 16 19:16:03.068222 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.068145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerStarted","Data":"6a8935db9bacda61eda514be408ee678c98f69cfc19174b67b6ceca27dbe7231"}
Sep 16 19:16:03.068436 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.068414    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:16:03.068565 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.068456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:16:03.069637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.069612    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:16:03.093441 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:03.093396    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podStartSLOduration=2.150906631 podStartE2EDuration="10.093383902s" totalImagesPullingTime="3.602457995s" totalInitContainerRuntime="4.340019276s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:03.091717647 +0000 UTC m=+1965.508420407" watchObservedRunningTime="2026-09-16 19:16:03.093383902 +0000 UTC m=+1965.510086661"
Sep 16 19:16:04.077473 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:04.077417    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:16:06.893617 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.893592    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:16:06.965601 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.965519    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmldj\" (UniqueName: \"kubernetes.io/projected/43e1002a-bc08-46e9-9094-8817ad346bae-kube-api-access-qmldj\") pod \"43e1002a-bc08-46e9-9094-8817ad346bae\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") "
Sep 16 19:16:06.965601 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.965575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls\") pod \"43e1002a-bc08-46e9-9094-8817ad346bae\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") "
Sep 16 19:16:06.965852 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.965610    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-kube-rbac-proxy-sar-config\") pod \"43e1002a-bc08-46e9-9094-8817ad346bae\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") "
Sep 16 19:16:06.965852 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.965745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-cabundle-cert\") pod \"43e1002a-bc08-46e9-9094-8817ad346bae\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") "
Sep 16 19:16:06.965852 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.965800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43e1002a-bc08-46e9-9094-8817ad346bae-kserve-provision-location\") pod \"43e1002a-bc08-46e9-9094-8817ad346bae\" (UID: \"43e1002a-bc08-46e9-9094-8817ad346bae\") "
Sep 16 19:16:06.966166 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.966088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43e1002a-bc08-46e9-9094-8817ad346bae" (UID: "43e1002a-bc08-46e9-9094-8817ad346bae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:06.966166 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.966095    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43e1002a-bc08-46e9-9094-8817ad346bae" (UID: "43e1002a-bc08-46e9-9094-8817ad346bae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:06.966166 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.966114    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e1002a-bc08-46e9-9094-8817ad346bae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43e1002a-bc08-46e9-9094-8817ad346bae" (UID: "43e1002a-bc08-46e9-9094-8817ad346bae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:06.967900 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.967877    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43e1002a-bc08-46e9-9094-8817ad346bae" (UID: "43e1002a-bc08-46e9-9094-8817ad346bae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:06.967992 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:06.967952    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e1002a-bc08-46e9-9094-8817ad346bae-kube-api-access-qmldj" (OuterVolumeSpecName: "kube-api-access-qmldj") pod "43e1002a-bc08-46e9-9094-8817ad346bae" (UID: "43e1002a-bc08-46e9-9094-8817ad346bae"). InnerVolumeSpecName "kube-api-access-qmldj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:07.066714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.066668    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmldj\" (UniqueName: \"kubernetes.io/projected/43e1002a-bc08-46e9-9094-8817ad346bae-kube-api-access-qmldj\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:16:07.066714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.066703    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43e1002a-bc08-46e9-9094-8817ad346bae-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:16:07.066714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.066714    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:16:07.066714 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.066722    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43e1002a-bc08-46e9-9094-8817ad346bae-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:16:07.066983 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.066730    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43e1002a-bc08-46e9-9094-8817ad346bae-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:16:07.086955 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.086922    2562 generic.go:356] "Generic (PLEG): container finished" podID="43e1002a-bc08-46e9-9094-8817ad346bae" containerID="e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b" exitCode=0
Sep 16 19:16:07.087096 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.086976    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"
Sep 16 19:16:07.087096 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.087004    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerDied","Data":"e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b"}
Sep 16 19:16:07.087096 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.087066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj" event={"ID":"43e1002a-bc08-46e9-9094-8817ad346bae","Type":"ContainerDied","Data":"08b50af1f672886ddad00b3188dc6e36eb8500e6f85820892fdfd4d99cc26249"}
Sep 16 19:16:07.087096 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.087091    2562 scope.go:117] "RemoveContainer" containerID="b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249"
Sep 16 19:16:07.095989 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.095971    2562 scope.go:117] "RemoveContainer" containerID="e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b"
Sep 16 19:16:07.104167 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.104147    2562 scope.go:117] "RemoveContainer" containerID="7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0"
Sep 16 19:16:07.110431 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.110401    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"]
Sep 16 19:16:07.112814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.112756    2562 scope.go:117] "RemoveContainer" containerID="b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249"
Sep 16 19:16:07.113262 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:16:07.113206    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249\": container with ID starting with b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249 not found: ID does not exist" containerID="b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249"
Sep 16 19:16:07.113403 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.113259    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249"} err="failed to get container status \"b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249\": rpc error: code = NotFound desc = could not find container \"b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249\": container with ID starting with b48ba244ac53aa60719688b46843a6c72399a63040b9cd834dfd42c6b3a80249 not found: ID does not exist"
Sep 16 19:16:07.113403 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.113277    2562 scope.go:117] "RemoveContainer" containerID="e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b"
Sep 16 19:16:07.113783 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:16:07.113763    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b\": container with ID starting with e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b not found: ID does not exist" containerID="e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b"
Sep 16 19:16:07.113873 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.113793    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b"} err="failed to get container status \"e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b\": rpc error: code = NotFound desc = could not find container \"e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b\": container with ID starting with e92a8113f7cc6794c8a21acc2a11705d65a202c3dfc7525622abeaac266b234b not found: ID does not exist"
Sep 16 19:16:07.113873 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.113816    2562 scope.go:117] "RemoveContainer" containerID="7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0"
Sep 16 19:16:07.114109 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:16:07.114085    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0\": container with ID starting with 7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0 not found: ID does not exist" containerID="7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0"
Sep 16 19:16:07.114164 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.114118    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0"} err="failed to get container status \"7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0\": rpc error: code = NotFound desc = could not find container \"7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0\": container with ID starting with 7136b0caecf0fff1efce9293ed92dc6547a514e7f6de9cca277f343de3568ab0 not found: ID does not exist"
Sep 16 19:16:07.115051 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:07.115033    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-b9b98bcd9-bj7dj"]
Sep 16 19:16:08.121737 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:08.121703    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" path="/var/lib/kubelet/pods/43e1002a-bc08-46e9-9094-8817ad346bae/volumes"
Sep 16 19:16:09.035713 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.035667    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:09.049079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049048    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"]
Sep 16 19:16:09.049412 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049399    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="storage-initializer"
Sep 16 19:16:09.049459 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049414    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="storage-initializer"
Sep 16 19:16:09.049459 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049424    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container"
Sep 16 19:16:09.049459 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049430    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container"
Sep 16 19:16:09.049459 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049449    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kube-rbac-proxy"
Sep 16 19:16:09.049459 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049455    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kube-rbac-proxy"
Sep 16 19:16:09.049620 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049500    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kube-rbac-proxy"
Sep 16 19:16:09.049620 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.049509    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e1002a-bc08-46e9-9094-8817ad346bae" containerName="kserve-container"
Sep 16 19:16:09.054534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.054515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.056829 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.056803    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:16:09.056947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.056823    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:16:09.062779 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.062759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"]
Sep 16 19:16:09.078656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.078636    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:16:09.079089 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.079068    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 16 19:16:09.189673 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.189639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4pk\" (UniqueName: \"kubernetes.io/projected/9b548cff-c012-4eb5-974f-44853c19992e-kube-api-access-kr4pk\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.190144 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.189898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b548cff-c012-4eb5-974f-44853c19992e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.190144 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.189933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b548cff-c012-4eb5-974f-44853c19992e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.190144 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.190064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.190279 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.190154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.291589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.291519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b548cff-c012-4eb5-974f-44853c19992e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.291589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.291553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b548cff-c012-4eb5-974f-44853c19992e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.291589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.291582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.291752 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.291616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.291752 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.291637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4pk\" (UniqueName: \"kubernetes.io/projected/9b548cff-c012-4eb5-974f-44853c19992e-kube-api-access-kr4pk\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.291930 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.291909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b548cff-c012-4eb5-974f-44853c19992e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.292358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.292305    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.292358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.292346    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.294144 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.294123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b548cff-c012-4eb5-974f-44853c19992e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.301164 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.301129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4pk\" (UniqueName: \"kubernetes.io/projected/9b548cff-c012-4eb5-974f-44853c19992e-kube-api-access-kr4pk\") pod \"isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.366047 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.366020    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:09.642376 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:09.642350    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"]
Sep 16 19:16:09.644291 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:16:09.644267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b548cff_c012_4eb5_974f_44853c19992e.slice/crio-95d8964ed7772f35da2c43c1be45ac0c22fc1c8a8e03f68387f6d8e7b9004588 WatchSource:0}: Error finding container 95d8964ed7772f35da2c43c1be45ac0c22fc1c8a8e03f68387f6d8e7b9004588: Status 404 returned error can't find the container with id 95d8964ed7772f35da2c43c1be45ac0c22fc1c8a8e03f68387f6d8e7b9004588
Sep 16 19:16:10.104630 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:10.104595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerStarted","Data":"7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6"}
Sep 16 19:16:10.104823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:10.104661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerStarted","Data":"95d8964ed7772f35da2c43c1be45ac0c22fc1c8a8e03f68387f6d8e7b9004588"}
Sep 16 19:16:14.123027 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:14.122993    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b548cff-c012-4eb5-974f-44853c19992e" containerID="7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6" exitCode=0
Sep 16 19:16:14.123487 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:14.123033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerDied","Data":"7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6"}
Sep 16 19:16:15.130171 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:15.130132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerStarted","Data":"fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033"}
Sep 16 19:16:15.130655 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:15.130210    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerStarted","Data":"edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5"}
Sep 16 19:16:15.130655 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:15.130447    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:15.150788 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:15.150743    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podStartSLOduration=2.502251418 podStartE2EDuration="6.15072365s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648472232s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:15.14911939 +0000 UTC m=+1977.565822140" watchObservedRunningTime="2026-09-16 19:16:15.15072365 +0000 UTC m=+1977.567426474"
Sep 16 19:16:16.134917 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:16.134876    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:16.136102 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:16.136077    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:17.140028 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:17.139985    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:19.079264 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:19.079237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:16:22.144268 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:22.144242    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:16:22.144755 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:22.144729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:32.144729 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:32.144688    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:34.820868 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:34.820838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"]
Sep 16 19:16:34.821295 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:34.821085    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" containerID="cri-o://1e5834cd6101beaa918ef1c5ab8fdedcb71b003bc27ef0e3b1bb88d1815ae3ff" gracePeriod=30
Sep 16 19:16:34.821295 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:34.821069    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container" containerID="cri-o://6a8935db9bacda61eda514be408ee678c98f69cfc19174b67b6ceca27dbe7231" gracePeriod=30
Sep 16 19:16:34.849204 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:34.849172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:35.217684 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:35.217596    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerID="1e5834cd6101beaa918ef1c5ab8fdedcb71b003bc27ef0e3b1bb88d1815ae3ff" exitCode=2
Sep 16 19:16:35.217684 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:35.217651    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerDied","Data":"1e5834cd6101beaa918ef1c5ab8fdedcb71b003bc27ef0e3b1bb88d1815ae3ff"}
Sep 16 19:16:39.075103 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:39.075063    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:16:39.979491 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:39.979456    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:40.304265 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:40.304225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:42.144868 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:42.144830    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:44.075221 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:44.075177    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:16:49.074852 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:49.074810    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:16:49.075298 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:49.074954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:16:52.145555 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:52.145517    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:16:54.074295 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:54.074249    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:16:59.074299 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:16:59.074254    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:17:02.145187 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:02.145142    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:17:04.074963 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:04.074913    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 16 19:17:05.348364 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.348306    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerID="6a8935db9bacda61eda514be408ee678c98f69cfc19174b67b6ceca27dbe7231" exitCode=137
Sep 16 19:17:05.348364 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.348353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerDied","Data":"6a8935db9bacda61eda514be408ee678c98f69cfc19174b67b6ceca27dbe7231"}
Sep 16 19:17:05.457247 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.457225    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:17:05.579029 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.578995    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2107cc-c3ae-4c2a-b416-1f5144608083-kserve-provision-location\") pod \"5c2107cc-c3ae-4c2a-b416-1f5144608083\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") "
Sep 16 19:17:05.579029 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579027    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xsnq\" (UniqueName: \"kubernetes.io/projected/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-api-access-6xsnq\") pod \"5c2107cc-c3ae-4c2a-b416-1f5144608083\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") "
Sep 16 19:17:05.579274 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579048    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-cabundle-cert\") pod \"5c2107cc-c3ae-4c2a-b416-1f5144608083\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") "
Sep 16 19:17:05.579274 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579084    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-rbac-proxy-sar-config\") pod \"5c2107cc-c3ae-4c2a-b416-1f5144608083\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") "
Sep 16 19:17:05.579274 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579218    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls\") pod \"5c2107cc-c3ae-4c2a-b416-1f5144608083\" (UID: \"5c2107cc-c3ae-4c2a-b416-1f5144608083\") "
Sep 16 19:17:05.579483 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579455    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c2107cc-c3ae-4c2a-b416-1f5144608083" (UID: "5c2107cc-c3ae-4c2a-b416-1f5144608083"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:05.579541 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579464    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c2107cc-c3ae-4c2a-b416-1f5144608083" (UID: "5c2107cc-c3ae-4c2a-b416-1f5144608083"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:05.579637 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579620    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:05.579695 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.579646    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:05.581353 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.581301    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-api-access-6xsnq" (OuterVolumeSpecName: "kube-api-access-6xsnq") pod "5c2107cc-c3ae-4c2a-b416-1f5144608083" (UID: "5c2107cc-c3ae-4c2a-b416-1f5144608083"). InnerVolumeSpecName "kube-api-access-6xsnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:05.581494 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.581474    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c2107cc-c3ae-4c2a-b416-1f5144608083" (UID: "5c2107cc-c3ae-4c2a-b416-1f5144608083"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:05.589419 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.589399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2107cc-c3ae-4c2a-b416-1f5144608083-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c2107cc-c3ae-4c2a-b416-1f5144608083" (UID: "5c2107cc-c3ae-4c2a-b416-1f5144608083"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:05.680691 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.680658    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c2107cc-c3ae-4c2a-b416-1f5144608083-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:05.680691 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.680687    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsnq\" (UniqueName: \"kubernetes.io/projected/5c2107cc-c3ae-4c2a-b416-1f5144608083-kube-api-access-6xsnq\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:05.680691 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:05.680697    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c2107cc-c3ae-4c2a-b416-1f5144608083-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:06.354702 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.354667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l" event={"ID":"5c2107cc-c3ae-4c2a-b416-1f5144608083","Type":"ContainerDied","Data":"496d32f7e55ebcae800b6fcae7356d0f037af1c8fe816ab9b6d55148dcc6d243"}
Sep 16 19:17:06.355085 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.354713    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"
Sep 16 19:17:06.355085 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.354738    2562 scope.go:117] "RemoveContainer" containerID="1e5834cd6101beaa918ef1c5ab8fdedcb71b003bc27ef0e3b1bb88d1815ae3ff"
Sep 16 19:17:06.363089 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.363071    2562 scope.go:117] "RemoveContainer" containerID="6a8935db9bacda61eda514be408ee678c98f69cfc19174b67b6ceca27dbe7231"
Sep 16 19:17:06.371063 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.371041    2562 scope.go:117] "RemoveContainer" containerID="0f875f7fb157a14be4760518c978de19716b957512f0561ee3833113190e533d"
Sep 16 19:17:06.376903 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.376880    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"]
Sep 16 19:17:06.381176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:06.381131    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-ssm8l"]
Sep 16 19:17:08.123797 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:08.123764    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" path="/var/lib/kubelet/pods/5c2107cc-c3ae-4c2a-b416-1f5144608083/volumes"
Sep 16 19:17:12.145002 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:12.144972    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:17:21.058870 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:21.058828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:22.144986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:22.144944    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:17:26.195822 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.195786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:26.230892 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.230861    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"]
Sep 16 19:17:26.231212 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231199    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="storage-initializer"
Sep 16 19:17:26.231255 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231214    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="storage-initializer"
Sep 16 19:17:26.231255 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231242    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy"
Sep 16 19:17:26.231255 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231248    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy"
Sep 16 19:17:26.231374 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231258    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container"
Sep 16 19:17:26.231374 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231263    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container"
Sep 16 19:17:26.231374 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231339    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kserve-container"
Sep 16 19:17:26.231374 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.231353    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2107cc-c3ae-4c2a-b416-1f5144608083" containerName="kube-rbac-proxy"
Sep 16 19:17:26.233679 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.233664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.235871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.235846    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 16 19:17:26.235996 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.235853    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 16 19:17:26.244612 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.244588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"]
Sep 16 19:17:26.357032 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.357002    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5v4h\" (UniqueName: \"kubernetes.io/projected/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-api-access-s5v4h\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.357205 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.357045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.357205 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.357065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.357205 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.357106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/248b68c0-53a3-4da0-90ef-22951081ea0b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.357366 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.357212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458223 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458223 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5v4h\" (UniqueName: \"kubernetes.io/projected/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-api-access-s5v4h\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458223 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458545 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458545 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/248b68c0-53a3-4da0-90ef-22951081ea0b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458545 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:17:26.458435    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 16 19:17:26.458545 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:17:26.458509    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls podName:248b68c0-53a3-4da0-90ef-22951081ea0b nodeName:}" failed. No retries permitted until 2026-09-16 19:17:26.95849374 +0000 UTC m=+2049.375196478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls") pod "isvc-triton-predictor-5d7db58676-gpgpq" (UID: "248b68c0-53a3-4da0-90ef-22951081ea0b") : secret "isvc-triton-predictor-serving-cert" not found
Sep 16 19:17:26.458726 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/248b68c0-53a3-4da0-90ef-22951081ea0b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.458995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.459030 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.458974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.467006 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.466988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5v4h\" (UniqueName: \"kubernetes.io/projected/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-api-access-s5v4h\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.637054 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.637019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:26.961753 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.961720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:26.964307 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:26.964283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-gpgpq\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:27.143380 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:27.143350    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:17:27.415003 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:27.414978    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"]
Sep 16 19:17:27.417688 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:17:27.417655    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248b68c0_53a3_4da0_90ef_22951081ea0b.slice/crio-45a7b9b54bb6761ca10cf82d15705e44283c101a07da34f9d4410e8b09842871 WatchSource:0}: Error finding container 45a7b9b54bb6761ca10cf82d15705e44283c101a07da34f9d4410e8b09842871: Status 404 returned error can't find the container with id 45a7b9b54bb6761ca10cf82d15705e44283c101a07da34f9d4410e8b09842871
Sep 16 19:17:27.447290 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:27.447264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerStarted","Data":"45a7b9b54bb6761ca10cf82d15705e44283c101a07da34f9d4410e8b09842871"}
Sep 16 19:17:28.453149 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:28.453107    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerStarted","Data":"566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d"}
Sep 16 19:17:29.525886 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:29.525851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.472675 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:31.472643    2562 generic.go:356] "Generic (PLEG): container finished" podID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerID="566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d" exitCode=0
Sep 16 19:17:31.473096 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:31.472689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerDied","Data":"566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d"}
Sep 16 19:17:32.145933 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:32.145905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:17:39.261480 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:39.256626    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"]
Sep 16 19:17:39.261480 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:39.256914    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" containerID="cri-o://edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5" gracePeriod=30
Sep 16 19:17:39.261480 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:39.257282    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kube-rbac-proxy" containerID="cri-o://fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033" gracePeriod=30
Sep 16 19:17:39.289736 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:39.289639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:40.051133 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:40.050831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:40.551147 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:40.551001    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b548cff-c012-4eb5-974f-44853c19992e" containerID="fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033" exitCode=2
Sep 16 19:17:40.551684 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:40.551153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerDied","Data":"fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033"}
Sep 16 19:17:42.140257 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:42.140190    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 16 19:17:42.144761 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:42.144727    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 16 19:17:44.380984 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.380881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:44.463711 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.463673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:44.513839 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.511816    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"]
Sep 16 19:17:44.517505 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.517478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.520802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.520478    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:17:44.520802 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.520674    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:17:44.534011 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.531946    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"]
Sep 16 19:17:44.537018 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.536865    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:44.643099 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.642758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b40722e-65f2-4428-b32c-9529b65207ae-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.643099 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.642847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49q6\" (UniqueName: \"kubernetes.io/projected/0b40722e-65f2-4428-b32c-9529b65207ae-kube-api-access-f49q6\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.643099 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.642877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b40722e-65f2-4428-b32c-9529b65207ae-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.643099 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.642913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.643099 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.642943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.744625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.743946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b40722e-65f2-4428-b32c-9529b65207ae-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.744625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.744033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f49q6\" (UniqueName: \"kubernetes.io/projected/0b40722e-65f2-4428-b32c-9529b65207ae-kube-api-access-f49q6\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.744625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.744063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b40722e-65f2-4428-b32c-9529b65207ae-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.744625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.744098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.744625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.744132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.746132 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.746094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b40722e-65f2-4428-b32c-9529b65207ae-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.746588 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.746562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.747203 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.747179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-cabundle-cert\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.758838 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.758793    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:44.759145 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.759120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49q6\" (UniqueName: \"kubernetes.io/projected/0b40722e-65f2-4428-b32c-9529b65207ae-kube-api-access-f49q6\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.760152 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.760129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b40722e-65f2-4428-b32c-9529b65207ae-proxy-tls\") pod \"isvc-canary-rollback-predictor-665c868b97-vqw4s\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:44.835480 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:44.835242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:45.132170 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.132075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"]
Sep 16 19:17:45.139533 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:17:45.139491    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b40722e_65f2_4428_b32c_9529b65207ae.slice/crio-1079b15aedac1a5b8370b9824c3bb54dea62a3a4bf67fab5e561aae5d79f686a WatchSource:0}: Error finding container 1079b15aedac1a5b8370b9824c3bb54dea62a3a4bf67fab5e561aae5d79f686a: Status 404 returned error can't find the container with id 1079b15aedac1a5b8370b9824c3bb54dea62a3a4bf67fab5e561aae5d79f686a
Sep 16 19:17:45.219808 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.219757    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:17:45.350564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.349956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b548cff-c012-4eb5-974f-44853c19992e-kserve-provision-location\") pod \"9b548cff-c012-4eb5-974f-44853c19992e\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") "
Sep 16 19:17:45.350564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.350061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-cabundle-cert\") pod \"9b548cff-c012-4eb5-974f-44853c19992e\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") "
Sep 16 19:17:45.350564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.350148    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr4pk\" (UniqueName: \"kubernetes.io/projected/9b548cff-c012-4eb5-974f-44853c19992e-kube-api-access-kr4pk\") pod \"9b548cff-c012-4eb5-974f-44853c19992e\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") "
Sep 16 19:17:45.350564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.350174    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b548cff-c012-4eb5-974f-44853c19992e-proxy-tls\") pod \"9b548cff-c012-4eb5-974f-44853c19992e\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") "
Sep 16 19:17:45.350564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.350207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-kube-rbac-proxy-sar-config\") pod \"9b548cff-c012-4eb5-974f-44853c19992e\" (UID: \"9b548cff-c012-4eb5-974f-44853c19992e\") "
Sep 16 19:17:45.350971 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.350789    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b548cff-c012-4eb5-974f-44853c19992e" (UID: "9b548cff-c012-4eb5-974f-44853c19992e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:45.351506 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.351441    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b548cff-c012-4eb5-974f-44853c19992e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b548cff-c012-4eb5-974f-44853c19992e" (UID: "9b548cff-c012-4eb5-974f-44853c19992e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:45.351506 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.351471    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b548cff-c012-4eb5-974f-44853c19992e" (UID: "9b548cff-c012-4eb5-974f-44853c19992e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:45.354076 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.354044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b548cff-c012-4eb5-974f-44853c19992e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b548cff-c012-4eb5-974f-44853c19992e" (UID: "9b548cff-c012-4eb5-974f-44853c19992e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:45.354076 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.354056    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b548cff-c012-4eb5-974f-44853c19992e-kube-api-access-kr4pk" (OuterVolumeSpecName: "kube-api-access-kr4pk") pod "9b548cff-c012-4eb5-974f-44853c19992e" (UID: "9b548cff-c012-4eb5-974f-44853c19992e"). InnerVolumeSpecName "kube-api-access-kr4pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:45.451823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.451720    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4pk\" (UniqueName: \"kubernetes.io/projected/9b548cff-c012-4eb5-974f-44853c19992e-kube-api-access-kr4pk\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:45.451823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.451756    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b548cff-c012-4eb5-974f-44853c19992e-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:45.451823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.451770    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:45.451823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.451782    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b548cff-c012-4eb5-974f-44853c19992e-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:45.451823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.451793    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b548cff-c012-4eb5-974f-44853c19992e-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:17:45.592309 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.592202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerStarted","Data":"d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214"}
Sep 16 19:17:45.592309 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.592281    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerStarted","Data":"1079b15aedac1a5b8370b9824c3bb54dea62a3a4bf67fab5e561aae5d79f686a"}
Sep 16 19:17:45.596114 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.596029    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b548cff-c012-4eb5-974f-44853c19992e" containerID="edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5" exitCode=0
Sep 16 19:17:45.596114 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.596081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerDied","Data":"edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5"}
Sep 16 19:17:45.596310 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.596142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x" event={"ID":"9b548cff-c012-4eb5-974f-44853c19992e","Type":"ContainerDied","Data":"95d8964ed7772f35da2c43c1be45ac0c22fc1c8a8e03f68387f6d8e7b9004588"}
Sep 16 19:17:45.596310 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.596092    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"
Sep 16 19:17:45.596310 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.596184    2562 scope.go:117] "RemoveContainer" containerID="fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033"
Sep 16 19:17:45.608551 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.607858    2562 scope.go:117] "RemoveContainer" containerID="edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5"
Sep 16 19:17:45.621203 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.621023    2562 scope.go:117] "RemoveContainer" containerID="7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6"
Sep 16 19:17:45.627342 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.627271    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"]
Sep 16 19:17:45.634872 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.634847    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7cc6db9b5d-vd84x"]
Sep 16 19:17:45.635505 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.635481    2562 scope.go:117] "RemoveContainer" containerID="fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033"
Sep 16 19:17:45.636123 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:17:45.635947    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033\": container with ID starting with fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033 not found: ID does not exist" containerID="fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033"
Sep 16 19:17:45.636123 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.635983    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033"} err="failed to get container status \"fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033\": rpc error: code = NotFound desc = could not find container \"fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033\": container with ID starting with fed5f5b4a8c188877c478f7d09d8a8cacd7c28e20cf00b749257ba41b790c033 not found: ID does not exist"
Sep 16 19:17:45.636123 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.636008    2562 scope.go:117] "RemoveContainer" containerID="edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5"
Sep 16 19:17:45.636381 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:17:45.636352    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5\": container with ID starting with edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5 not found: ID does not exist" containerID="edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5"
Sep 16 19:17:45.636485 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.636385    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5"} err="failed to get container status \"edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5\": rpc error: code = NotFound desc = could not find container \"edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5\": container with ID starting with edb8a73d0e5184a2c156007ea30636ecadf0269734bb71595d897d7da1ca24f5 not found: ID does not exist"
Sep 16 19:17:45.636485 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.636409    2562 scope.go:117] "RemoveContainer" containerID="7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6"
Sep 16 19:17:45.636798 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:17:45.636742    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6\": container with ID starting with 7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6 not found: ID does not exist" containerID="7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6"
Sep 16 19:17:45.636798 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:45.636774    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6"} err="failed to get container status \"7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6\": rpc error: code = NotFound desc = could not find container \"7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6\": container with ID starting with 7b9f22cc2f42317316f2c264ddce4942678a09763aa9b29ee017845c351b1eb6 not found: ID does not exist"
Sep 16 19:17:46.125667 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:46.125186    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b548cff-c012-4eb5-974f-44853c19992e" path="/var/lib/kubelet/pods/9b548cff-c012-4eb5-974f-44853c19992e/volumes"
Sep 16 19:17:50.632381 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:50.632311    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b40722e-65f2-4428-b32c-9529b65207ae" containerID="d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214" exitCode=0
Sep 16 19:17:50.632848 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:50.632457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerDied","Data":"d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214"}
Sep 16 19:17:51.641608 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:51.641569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerStarted","Data":"5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751"}
Sep 16 19:17:51.642185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:51.641647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerStarted","Data":"e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be"}
Sep 16 19:17:51.642185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:51.641971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:51.642185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:51.642158    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:51.643444 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:51.643409    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:17:51.663889 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:51.663823    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podStartSLOduration=3.282961057 podStartE2EDuration="7.6638025s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.380841443s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:51.662497481 +0000 UTC m=+2074.079200252" watchObservedRunningTime="2026-09-16 19:17:51.6638025 +0000 UTC m=+2074.080505261"
Sep 16 19:17:52.647787 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:52.647748    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:17:55.208762 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:55.208692    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:57.653815 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:57.653780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:17:57.654454 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:17:57.654419    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:07.654473 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:18:07.654416    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:17.655397 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:18:17.655349    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:27.654890 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:18:27.654845    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:37.655156 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:18:37.655118    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:47.654920 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:18:47.654871    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:18:57.655419 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:18:57.655378    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:19:01.111745 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:01.111708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:01.216250 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:01.216208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:07.654546 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:07.654497    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:19:17.654648 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:17.654546    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:19:25.118399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:25.118369    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:19:30.151032 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:30.151006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerStarted","Data":"4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e"}
Sep 16 19:19:31.157294 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:31.157256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerStarted","Data":"865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10"}
Sep 16 19:19:31.157821 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:31.157362    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:19:31.176658 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:31.176611    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" podStartSLOduration=2.875526615 podStartE2EDuration="2m5.176596525s" totalImagesPullingTime="1m58.544349525s" totalInitContainerRuntime="3.756720385s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:31.175415878 +0000 UTC m=+2173.592118639" watchObservedRunningTime="2026-09-16 19:19:31.176596525 +0000 UTC m=+2173.593299283"
Sep 16 19:19:31.325594 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:31.325558    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:31.385077 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:31.385040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:31.641171 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:31.641133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:32.161484 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:32.161453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:19:32.162564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:32.162535    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:19:33.165910 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:33.165868    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 16 19:19:34.548567 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:34.548529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:38.171641 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:38.171614    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:19:38.172343 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:38.172306    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:19:48.527284 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:48.527251    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"]
Sep 16 19:19:48.527826 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:48.527539    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kserve-container" containerID="cri-o://4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e" gracePeriod=30
Sep 16 19:19:48.527826 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:48.527572    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kube-rbac-proxy" containerID="cri-o://865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10" gracePeriod=30
Sep 16 19:19:48.543871 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:48.543836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:49.231824 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:49.231790    2562 generic.go:356] "Generic (PLEG): container finished" podID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerID="865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10" exitCode=2
Sep 16 19:19:49.231997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:49.231863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerDied","Data":"865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10"}
Sep 16 19:19:50.898730 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:50.898707    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:19:51.015006 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.014922    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls\") pod \"248b68c0-53a3-4da0-90ef-22951081ea0b\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") "
Sep 16 19:19:51.015154 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015009    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5v4h\" (UniqueName: \"kubernetes.io/projected/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-api-access-s5v4h\") pod \"248b68c0-53a3-4da0-90ef-22951081ea0b\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") "
Sep 16 19:19:51.015196 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-cabundle-cert\") pod \"248b68c0-53a3-4da0-90ef-22951081ea0b\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") "
Sep 16 19:19:51.015229 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015214    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-rbac-proxy-sar-config\") pod \"248b68c0-53a3-4da0-90ef-22951081ea0b\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") "
Sep 16 19:19:51.015369 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/248b68c0-53a3-4da0-90ef-22951081ea0b-kserve-provision-location\") pod \"248b68c0-53a3-4da0-90ef-22951081ea0b\" (UID: \"248b68c0-53a3-4da0-90ef-22951081ea0b\") "
Sep 16 19:19:51.015665 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015564    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "248b68c0-53a3-4da0-90ef-22951081ea0b" (UID: "248b68c0-53a3-4da0-90ef-22951081ea0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:51.015946 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "248b68c0-53a3-4da0-90ef-22951081ea0b" (UID: "248b68c0-53a3-4da0-90ef-22951081ea0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:51.015946 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.015818    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248b68c0-53a3-4da0-90ef-22951081ea0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "248b68c0-53a3-4da0-90ef-22951081ea0b" (UID: "248b68c0-53a3-4da0-90ef-22951081ea0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:51.017751 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.017733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "248b68c0-53a3-4da0-90ef-22951081ea0b" (UID: "248b68c0-53a3-4da0-90ef-22951081ea0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:51.017826 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.017775    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-api-access-s5v4h" (OuterVolumeSpecName: "kube-api-access-s5v4h") pod "248b68c0-53a3-4da0-90ef-22951081ea0b" (UID: "248b68c0-53a3-4da0-90ef-22951081ea0b"). InnerVolumeSpecName "kube-api-access-s5v4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:51.116698 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.116666    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5v4h\" (UniqueName: \"kubernetes.io/projected/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-api-access-s5v4h\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.116698 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.116693    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.116698 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.116703    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/248b68c0-53a3-4da0-90ef-22951081ea0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.116929 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.116713    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/248b68c0-53a3-4da0-90ef-22951081ea0b-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.116929 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.116721    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/248b68c0-53a3-4da0-90ef-22951081ea0b-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:19:51.245691 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.245660    2562 generic.go:356] "Generic (PLEG): container finished" podID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerID="4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e" exitCode=0
Sep 16 19:19:51.245859 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.245716    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"
Sep 16 19:19:51.245859 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.245734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerDied","Data":"4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e"}
Sep 16 19:19:51.245859 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.245790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq" event={"ID":"248b68c0-53a3-4da0-90ef-22951081ea0b","Type":"ContainerDied","Data":"45a7b9b54bb6761ca10cf82d15705e44283c101a07da34f9d4410e8b09842871"}
Sep 16 19:19:51.245859 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.245817    2562 scope.go:117] "RemoveContainer" containerID="865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10"
Sep 16 19:19:51.255008 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.254991    2562 scope.go:117] "RemoveContainer" containerID="4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e"
Sep 16 19:19:51.262627 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.262612    2562 scope.go:117] "RemoveContainer" containerID="566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d"
Sep 16 19:19:51.270769 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.270715    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"]
Sep 16 19:19:51.271564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.271479    2562 scope.go:117] "RemoveContainer" containerID="865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10"
Sep 16 19:19:51.272008 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:19:51.271982    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10\": container with ID starting with 865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10 not found: ID does not exist" containerID="865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10"
Sep 16 19:19:51.272103 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.272013    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10"} err="failed to get container status \"865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10\": rpc error: code = NotFound desc = could not find container \"865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10\": container with ID starting with 865045366df3c11630d52ded1b8fa8cd1d619633b686fa3e376e6ed057782c10 not found: ID does not exist"
Sep 16 19:19:51.272103 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.272037    2562 scope.go:117] "RemoveContainer" containerID="4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e"
Sep 16 19:19:51.272373 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:19:51.272354    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e\": container with ID starting with 4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e not found: ID does not exist" containerID="4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e"
Sep 16 19:19:51.272432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.272378    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e"} err="failed to get container status \"4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e\": rpc error: code = NotFound desc = could not find container \"4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e\": container with ID starting with 4df48a22b37f2a00fb72b0fb473d94adf9890c0563bdaa6df78319d4772e3f1e not found: ID does not exist"
Sep 16 19:19:51.272432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.272394    2562 scope.go:117] "RemoveContainer" containerID="566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d"
Sep 16 19:19:51.272667 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:19:51.272652    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d\": container with ID starting with 566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d not found: ID does not exist" containerID="566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d"
Sep 16 19:19:51.272740 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.272670    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d"} err="failed to get container status \"566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d\": rpc error: code = NotFound desc = could not find container \"566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d\": container with ID starting with 566d2cff89237c6ae55cedd49fe5347bf51c1fd5518e2cd0f73dda0e90bbbe6d not found: ID does not exist"
Sep 16 19:19:51.273562 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:51.273544    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-gpgpq"]
Sep 16 19:19:52.122304 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:52.122275    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" path="/var/lib/kubelet/pods/248b68c0-53a3-4da0-90ef-22951081ea0b/volumes"
Sep 16 19:19:53.696415 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:53.696377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:53.829010 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:19:53.828974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:02.552780 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:02.552734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:04.109881 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:04.109851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:05.917604 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:05.917562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:14.693766 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:14.693731    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"]
Sep 16 19:21:14.694180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:14.693963    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" containerID="cri-o://e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be" gracePeriod=30
Sep 16 19:21:14.694180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:14.694016    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kube-rbac-proxy" containerID="cri-o://5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751" gracePeriod=30
Sep 16 19:21:14.702945 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:14.702916    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:15.117709 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:15.117661    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 16 19:21:15.612945 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:15.612910    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b40722e-65f2-4428-b32c-9529b65207ae" containerID="5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751" exitCode=2
Sep 16 19:21:15.613128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:15.612950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerDied","Data":"5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751"}
Sep 16 19:21:17.648252 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:17.648215    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 16 19:21:19.242509 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.242487    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:21:19.354061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.353994    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f49q6\" (UniqueName: \"kubernetes.io/projected/0b40722e-65f2-4428-b32c-9529b65207ae-kube-api-access-f49q6\") pod \"0b40722e-65f2-4428-b32c-9529b65207ae\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") "
Sep 16 19:21:19.354061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-cabundle-cert\") pod \"0b40722e-65f2-4428-b32c-9529b65207ae\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") "
Sep 16 19:21:19.354061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-kube-rbac-proxy-sar-config\") pod \"0b40722e-65f2-4428-b32c-9529b65207ae\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") "
Sep 16 19:21:19.354283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354087    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b40722e-65f2-4428-b32c-9529b65207ae-kserve-provision-location\") pod \"0b40722e-65f2-4428-b32c-9529b65207ae\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") "
Sep 16 19:21:19.354283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b40722e-65f2-4428-b32c-9529b65207ae-proxy-tls\") pod \"0b40722e-65f2-4428-b32c-9529b65207ae\" (UID: \"0b40722e-65f2-4428-b32c-9529b65207ae\") "
Sep 16 19:21:19.354447 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b40722e-65f2-4428-b32c-9529b65207ae" (UID: "0b40722e-65f2-4428-b32c-9529b65207ae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:19.354539 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354455    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b40722e-65f2-4428-b32c-9529b65207ae" (UID: "0b40722e-65f2-4428-b32c-9529b65207ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:19.354539 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.354466    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b40722e-65f2-4428-b32c-9529b65207ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b40722e-65f2-4428-b32c-9529b65207ae" (UID: "0b40722e-65f2-4428-b32c-9529b65207ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:19.356190 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.356170    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b40722e-65f2-4428-b32c-9529b65207ae-kube-api-access-f49q6" (OuterVolumeSpecName: "kube-api-access-f49q6") pod "0b40722e-65f2-4428-b32c-9529b65207ae" (UID: "0b40722e-65f2-4428-b32c-9529b65207ae"). InnerVolumeSpecName "kube-api-access-f49q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:19.356359 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.356300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b40722e-65f2-4428-b32c-9529b65207ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b40722e-65f2-4428-b32c-9529b65207ae" (UID: "0b40722e-65f2-4428-b32c-9529b65207ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:19.455155 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.455124    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f49q6\" (UniqueName: \"kubernetes.io/projected/0b40722e-65f2-4428-b32c-9529b65207ae-kube-api-access-f49q6\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.455155 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.455149    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.455155 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.455161    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b40722e-65f2-4428-b32c-9529b65207ae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.455394 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.455171    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b40722e-65f2-4428-b32c-9529b65207ae-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.455394 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.455180    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b40722e-65f2-4428-b32c-9529b65207ae-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:21:19.632763 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.632686    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b40722e-65f2-4428-b32c-9529b65207ae" containerID="e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be" exitCode=0
Sep 16 19:21:19.632763 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.632730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerDied","Data":"e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be"}
Sep 16 19:21:19.632763 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.632741    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"
Sep 16 19:21:19.632990 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.632779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s" event={"ID":"0b40722e-65f2-4428-b32c-9529b65207ae","Type":"ContainerDied","Data":"1079b15aedac1a5b8370b9824c3bb54dea62a3a4bf67fab5e561aae5d79f686a"}
Sep 16 19:21:19.632990 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.632805    2562 scope.go:117] "RemoveContainer" containerID="5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751"
Sep 16 19:21:19.641812 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.641794    2562 scope.go:117] "RemoveContainer" containerID="e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be"
Sep 16 19:21:19.649262 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.649243    2562 scope.go:117] "RemoveContainer" containerID="d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214"
Sep 16 19:21:19.654809 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.654786    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"]
Sep 16 19:21:19.656943 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.656923    2562 scope.go:117] "RemoveContainer" containerID="5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751"
Sep 16 19:21:19.657219 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:21:19.657199    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751\": container with ID starting with 5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751 not found: ID does not exist" containerID="5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751"
Sep 16 19:21:19.657271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.657228    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751"} err="failed to get container status \"5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751\": rpc error: code = NotFound desc = could not find container \"5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751\": container with ID starting with 5b5da6b4381a38d17a9e68a2f68d826ab029a76d8f5e2284de2d0427c76c6751 not found: ID does not exist"
Sep 16 19:21:19.657271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.657245    2562 scope.go:117] "RemoveContainer" containerID="e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be"
Sep 16 19:21:19.657530 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:21:19.657514    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be\": container with ID starting with e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be not found: ID does not exist" containerID="e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be"
Sep 16 19:21:19.657572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.657537    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be"} err="failed to get container status \"e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be\": rpc error: code = NotFound desc = could not find container \"e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be\": container with ID starting with e3c169c6d13c684fd02e190102d26fb70094e50b31ab1d67e893080753d3d5be not found: ID does not exist"
Sep 16 19:21:19.657572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.657553    2562 scope.go:117] "RemoveContainer" containerID="d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214"
Sep 16 19:21:19.657769 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:21:19.657748    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214\": container with ID starting with d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214 not found: ID does not exist" containerID="d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214"
Sep 16 19:21:19.657824 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.657778    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214"} err="failed to get container status \"d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214\": rpc error: code = NotFound desc = could not find container \"d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214\": container with ID starting with d02426c9f3517bfe08ded194fecc7d763da702b376b400d1fbd96ae4465ad214 not found: ID does not exist"
Sep 16 19:21:19.660927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.660908    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-665c868b97-vqw4s"]
Sep 16 19:21:19.828341 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.828277    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:19.913261 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.913173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:19.957579 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957546    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"]
Sep 16 19:21:19.957881 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957869    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="storage-initializer"
Sep 16 19:21:19.957927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957883    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="storage-initializer"
Sep 16 19:21:19.957927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957893    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container"
Sep 16 19:21:19.957927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957899    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container"
Sep 16 19:21:19.957927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957913    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="storage-initializer"
Sep 16 19:21:19.957927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957919    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="storage-initializer"
Sep 16 19:21:19.957927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957925    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957931    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957938    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957943    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957952    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957956    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957965    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957970    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957976    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="storage-initializer"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957981    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="storage-initializer"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957987    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.957992    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.958037    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.958045    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.958050    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.958055    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b548cff-c012-4eb5-974f-44853c19992e" containerName="kube-rbac-proxy"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.958062    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" containerName="kserve-container"
Sep 16 19:21:19.958105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.958066    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="248b68c0-53a3-4da0-90ef-22951081ea0b" containerName="kube-rbac-proxy"
Sep 16 19:21:19.960835 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.960819    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:19.963233 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.963204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:21:19.963388 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.963301    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:21:19.963440 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.963385    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:21:19.963584 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.963568    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:21:19.963653 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.963629    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:21:19.963713 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.963655    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:21:19.964153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.964138    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:21:19.981000 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.980975    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"]
Sep 16 19:21:19.996494 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:19.996469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:20.059702 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.059672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f72\" (UniqueName: \"kubernetes.io/projected/da51a222-ff7d-462e-86b0-bf74060d4363-kube-api-access-h9f72\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.059842 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.059715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.059842 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.059740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.059842 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.059758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.059842 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.059776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da51a222-ff7d-462e-86b0-bf74060d4363-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.121768 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.121735    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b40722e-65f2-4428-b32c-9529b65207ae" path="/var/lib/kubelet/pods/0b40722e-65f2-4428-b32c-9529b65207ae/volumes"
Sep 16 19:21:20.160769 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.160749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.160835 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.160780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.160835 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.160798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.160835 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.160815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da51a222-ff7d-462e-86b0-bf74060d4363-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.161027 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.160886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f72\" (UniqueName: \"kubernetes.io/projected/da51a222-ff7d-462e-86b0-bf74060d4363-kube-api-access-h9f72\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.161027 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:21:20.160912    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:21:20.161027 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:21:20.160967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls podName:da51a222-ff7d-462e-86b0-bf74060d4363 nodeName:}" failed. No retries permitted until 2026-09-16 19:21:20.660951187 +0000 UTC m=+2283.077653924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls") pod "isvc-canary-stop-predictor-7bb9c548b5-xr4pd" (UID: "da51a222-ff7d-462e-86b0-bf74060d4363") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 16 19:21:20.161259 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.161239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da51a222-ff7d-462e-86b0-bf74060d4363-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.161420 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.161404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.161513 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.161498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-cabundle-cert\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.177980 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.177920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f72\" (UniqueName: \"kubernetes.io/projected/da51a222-ff7d-462e-86b0-bf74060d4363-kube-api-access-h9f72\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.233837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.233809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:20.664662 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.664616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.667382 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.667358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls\") pod \"isvc-canary-stop-predictor-7bb9c548b5-xr4pd\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:20.871438 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:20.871386    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:21.145682 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:21.145659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"]
Sep 16 19:21:21.148250 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:21:21.148224    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda51a222_ff7d_462e_86b0_bf74060d4363.slice/crio-778c653e60c97f192d9049fa8034436ab16c3e5d4c4557e2607a6086f822fd74 WatchSource:0}: Error finding container 778c653e60c97f192d9049fa8034436ab16c3e5d4c4557e2607a6086f822fd74: Status 404 returned error can't find the container with id 778c653e60c97f192d9049fa8034436ab16c3e5d4c4557e2607a6086f822fd74
Sep 16 19:21:21.644877 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:21.644832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerStarted","Data":"b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0"}
Sep 16 19:21:21.645046 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:21.644897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerStarted","Data":"778c653e60c97f192d9049fa8034436ab16c3e5d4c4557e2607a6086f822fd74"}
Sep 16 19:21:23.848482 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:23.848449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:25.667896 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:25.667864    2562 generic.go:356] "Generic (PLEG): container finished" podID="da51a222-ff7d-462e-86b0-bf74060d4363" containerID="b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0" exitCode=0
Sep 16 19:21:25.668247 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:25.667911    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerDied","Data":"b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0"}
Sep 16 19:21:26.675252 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:26.675222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerStarted","Data":"0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2"}
Sep 16 19:21:26.675656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:26.675280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerStarted","Data":"b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5"}
Sep 16 19:21:26.675656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:26.675500    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:26.696092 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:26.696034    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podStartSLOduration=3.860420944 podStartE2EDuration="7.696016076s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835595132s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:26.695819272 +0000 UTC m=+2289.112522032" watchObservedRunningTime="2026-09-16 19:21:26.696016076 +0000 UTC m=+2289.112718837"
Sep 16 19:21:27.679489 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:27.679453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:27.680814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:27.680784    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:21:28.683169 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:28.683126    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:21:28.992348 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:28.992236    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:29.018406 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.018371    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"]
Sep 16 19:21:29.020872 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.020853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.024172 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.024147    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:21:29.024718 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.024694    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:21:29.033810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.033787    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"]
Sep 16 19:21:29.094108 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.094075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:29.138269 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.138235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hb6\" (UniqueName: \"kubernetes.io/projected/02bb16bc-7304-4345-82c1-182ce5651130-kube-api-access-h5hb6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.138454 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.138276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02bb16bc-7304-4345-82c1-182ce5651130-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.138454 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.138300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02bb16bc-7304-4345-82c1-182ce5651130-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.138454 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.138425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.138622 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.138466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.239382 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.239347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hb6\" (UniqueName: \"kubernetes.io/projected/02bb16bc-7304-4345-82c1-182ce5651130-kube-api-access-h5hb6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.239531 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.239387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02bb16bc-7304-4345-82c1-182ce5651130-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.239531 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.239410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02bb16bc-7304-4345-82c1-182ce5651130-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.239531 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.239442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.239644 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.239535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.239827 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.239809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02bb16bc-7304-4345-82c1-182ce5651130-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.240285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.240263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.240393 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.240364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.242105 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.242083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02bb16bc-7304-4345-82c1-182ce5651130-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.249206 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.249149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hb6\" (UniqueName: \"kubernetes.io/projected/02bb16bc-7304-4345-82c1-182ce5651130-kube-api-access-h5hb6\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.331035 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.331003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:29.603564 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.603532    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"]
Sep 16 19:21:29.604855 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:21:29.604829    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bb16bc_7304_4345_82c1_182ce5651130.slice/crio-beafe104fcd31b15c156c94778ad729f66d1b900655058d596b389db8271c91e WatchSource:0}: Error finding container beafe104fcd31b15c156c94778ad729f66d1b900655058d596b389db8271c91e: Status 404 returned error can't find the container with id beafe104fcd31b15c156c94778ad729f66d1b900655058d596b389db8271c91e
Sep 16 19:21:29.695123 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.695085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerStarted","Data":"b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2"}
Sep 16 19:21:29.695542 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:29.695186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerStarted","Data":"beafe104fcd31b15c156c94778ad729f66d1b900655058d596b389db8271c91e"}
Sep 16 19:21:33.688197 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:33.688164    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:21:33.688869 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:33.688840    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:21:33.715778 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:33.715743    2562 generic.go:356] "Generic (PLEG): container finished" podID="02bb16bc-7304-4345-82c1-182ce5651130" containerID="b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2" exitCode=0
Sep 16 19:21:33.715945 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:33.715790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerDied","Data":"b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2"}
Sep 16 19:21:34.722590 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:34.722550    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerStarted","Data":"00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9"}
Sep 16 19:21:34.722960 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:34.722620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerStarted","Data":"93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36"}
Sep 16 19:21:34.722960 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:34.722807    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:34.758389 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:34.758336    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" podStartSLOduration=2.857069492 podStartE2EDuration="6.758303633s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901234141s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:34.755373957 +0000 UTC m=+2297.172076717" watchObservedRunningTime="2026-09-16 19:21:34.758303633 +0000 UTC m=+2297.175006392"
Sep 16 19:21:35.726873 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:35.726845    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:41.736899 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:41.736869    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:21:43.689573 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:43.689534    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:21:53.689490 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:21:53.689452    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:22:03.689310 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:03.689268    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:22:11.741010 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:11.740927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:22:13.688880 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:13.688839    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:22:19.075504 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:19.075466    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"]
Sep 16 19:22:19.075936 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:19.075773    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kserve-container" containerID="cri-o://93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36" gracePeriod=30
Sep 16 19:22:19.075936 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:19.075808    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kube-rbac-proxy" containerID="cri-o://00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9" gracePeriod=30
Sep 16 19:22:19.110030 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:19.109999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:19.925235 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:19.925202    2562 generic.go:356] "Generic (PLEG): container finished" podID="02bb16bc-7304-4345-82c1-182ce5651130" containerID="00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9" exitCode=2
Sep 16 19:22:19.925443 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:19.925273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerDied","Data":"00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9"}
Sep 16 19:22:21.732259 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:21.732221    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 16 19:22:21.737658 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:21.737633    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 16 19:22:23.689555 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:23.689519    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:22:24.224079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.224041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:24.259445 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.259410    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"]
Sep 16 19:22:24.261992 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.261976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.264309 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.264287    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:22:24.264436 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.264287    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:22:24.272739 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.272717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"]
Sep 16 19:22:24.430961 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.430931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dsb\" (UniqueName: \"kubernetes.io/projected/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-api-access-c8dsb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.431153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.430986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90619afa-84ae-4550-abd4-6cdbeddbce49-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.431153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.431083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.431153 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.431121    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.431273 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.431188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.531795 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.531711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90619afa-84ae-4550-abd4-6cdbeddbce49-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.531795 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.531757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.531795 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.531780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.532095 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.531809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.532095 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:24.531894    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:22:24.532095 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:24.531960    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls podName:90619afa-84ae-4550-abd4-6cdbeddbce49 nodeName:}" failed. No retries permitted until 2026-09-16 19:22:25.031941306 +0000 UTC m=+2347.448644046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" (UID: "90619afa-84ae-4550-abd4-6cdbeddbce49") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:22:24.532095 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.532013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dsb\" (UniqueName: \"kubernetes.io/projected/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-api-access-c8dsb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.532263 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.532126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90619afa-84ae-4550-abd4-6cdbeddbce49-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.532533 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.532515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.532588 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.532517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.542544 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.542517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dsb\" (UniqueName: \"kubernetes.io/projected/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-api-access-c8dsb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:24.771389 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:24.771351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:25.037730 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.037686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:25.040201 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.040171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d5prm\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:25.171874 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.171838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:25.444687 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.444661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"]
Sep 16 19:22:25.465341 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:22:25.465290    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90619afa_84ae_4550_abd4_6cdbeddbce49.slice/crio-f264e80a5a36e92418208dce363d40a0343f28d25a1f2152ce9e5afdf36f0cae WatchSource:0}: Error finding container f264e80a5a36e92418208dce363d40a0343f28d25a1f2152ce9e5afdf36f0cae: Status 404 returned error can't find the container with id f264e80a5a36e92418208dce363d40a0343f28d25a1f2152ce9e5afdf36f0cae
Sep 16 19:22:25.620759 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.620736    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:22:25.743128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743042    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02bb16bc-7304-4345-82c1-182ce5651130-proxy-tls\") pod \"02bb16bc-7304-4345-82c1-182ce5651130\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") "
Sep 16 19:22:25.743128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743091    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02bb16bc-7304-4345-82c1-182ce5651130-kserve-provision-location\") pod \"02bb16bc-7304-4345-82c1-182ce5651130\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") "
Sep 16 19:22:25.743128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743123    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5hb6\" (UniqueName: \"kubernetes.io/projected/02bb16bc-7304-4345-82c1-182ce5651130-kube-api-access-h5hb6\") pod \"02bb16bc-7304-4345-82c1-182ce5651130\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") "
Sep 16 19:22:25.743457 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-kube-rbac-proxy-sar-config\") pod \"02bb16bc-7304-4345-82c1-182ce5651130\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") "
Sep 16 19:22:25.743457 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-cabundle-cert\") pod \"02bb16bc-7304-4345-82c1-182ce5651130\" (UID: \"02bb16bc-7304-4345-82c1-182ce5651130\") "
Sep 16 19:22:25.743457 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bb16bc-7304-4345-82c1-182ce5651130-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02bb16bc-7304-4345-82c1-182ce5651130" (UID: "02bb16bc-7304-4345-82c1-182ce5651130"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:25.743625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743614    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02bb16bc-7304-4345-82c1-182ce5651130-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.743728 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743672    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02bb16bc-7304-4345-82c1-182ce5651130" (UID: "02bb16bc-7304-4345-82c1-182ce5651130"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:25.743728 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.743692    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02bb16bc-7304-4345-82c1-182ce5651130" (UID: "02bb16bc-7304-4345-82c1-182ce5651130"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:25.745334 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.745290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bb16bc-7304-4345-82c1-182ce5651130-kube-api-access-h5hb6" (OuterVolumeSpecName: "kube-api-access-h5hb6") pod "02bb16bc-7304-4345-82c1-182ce5651130" (UID: "02bb16bc-7304-4345-82c1-182ce5651130"). InnerVolumeSpecName "kube-api-access-h5hb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:25.745444 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.745424    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02bb16bc-7304-4345-82c1-182ce5651130-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02bb16bc-7304-4345-82c1-182ce5651130" (UID: "02bb16bc-7304-4345-82c1-182ce5651130"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:25.844532 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.844498    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02bb16bc-7304-4345-82c1-182ce5651130-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.844532 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.844523    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hb6\" (UniqueName: \"kubernetes.io/projected/02bb16bc-7304-4345-82c1-182ce5651130-kube-api-access-h5hb6\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.844532 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.844534    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.844947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.844543    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02bb16bc-7304-4345-82c1-182ce5651130-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.958470 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.958432    2562 generic.go:356] "Generic (PLEG): container finished" podID="02bb16bc-7304-4345-82c1-182ce5651130" containerID="93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36" exitCode=0
Sep 16 19:22:25.958657 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.958489    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"
Sep 16 19:22:25.958657 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.958508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerDied","Data":"93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36"}
Sep 16 19:22:25.958657 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.958572    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj" event={"ID":"02bb16bc-7304-4345-82c1-182ce5651130","Type":"ContainerDied","Data":"beafe104fcd31b15c156c94778ad729f66d1b900655058d596b389db8271c91e"}
Sep 16 19:22:25.958657 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.958611    2562 scope.go:117] "RemoveContainer" containerID="00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9"
Sep 16 19:22:25.960463 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.960430    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerStarted","Data":"92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503"}
Sep 16 19:22:25.960610 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.960499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerStarted","Data":"f264e80a5a36e92418208dce363d40a0343f28d25a1f2152ce9e5afdf36f0cae"}
Sep 16 19:22:25.968512 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.968493    2562 scope.go:117] "RemoveContainer" containerID="93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36"
Sep 16 19:22:25.977816 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.977797    2562 scope.go:117] "RemoveContainer" containerID="b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2"
Sep 16 19:22:25.986900 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.986877    2562 scope.go:117] "RemoveContainer" containerID="00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9"
Sep 16 19:22:25.987222 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:25.987201    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9\": container with ID starting with 00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9 not found: ID does not exist" containerID="00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9"
Sep 16 19:22:25.987308 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.987229    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9"} err="failed to get container status \"00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9\": rpc error: code = NotFound desc = could not find container \"00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9\": container with ID starting with 00aa47182747de69cc443cfaef1de967d613cadfeceecda02a840339a895aad9 not found: ID does not exist"
Sep 16 19:22:25.987308 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.987247    2562 scope.go:117] "RemoveContainer" containerID="93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36"
Sep 16 19:22:25.987529 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:25.987510    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36\": container with ID starting with 93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36 not found: ID does not exist" containerID="93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36"
Sep 16 19:22:25.987598 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.987542    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36"} err="failed to get container status \"93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36\": rpc error: code = NotFound desc = could not find container \"93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36\": container with ID starting with 93c0109a318c2e2dcfbf703f223190e37498efde8f27218ce874b8f8692c8c36 not found: ID does not exist"
Sep 16 19:22:25.987598 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.987564    2562 scope.go:117] "RemoveContainer" containerID="b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2"
Sep 16 19:22:25.987821 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:25.987804    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2\": container with ID starting with b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2 not found: ID does not exist" containerID="b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2"
Sep 16 19:22:25.987872 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.987828    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2"} err="failed to get container status \"b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2\": rpc error: code = NotFound desc = could not find container \"b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2\": container with ID starting with b590a9349069afabde3ec28595b945a78777618550cbdc67f15add42d167c2a2 not found: ID does not exist"
Sep 16 19:22:25.997591 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:25.997529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"]
Sep 16 19:22:26.004466 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:26.004439    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-9r8tj"]
Sep 16 19:22:26.123784 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:26.123753    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02bb16bc-7304-4345-82c1-182ce5651130" path="/var/lib/kubelet/pods/02bb16bc-7304-4345-82c1-182ce5651130/volumes"
Sep 16 19:22:29.979205 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:29.979173    2562 generic.go:356] "Generic (PLEG): container finished" podID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerID="92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503" exitCode=0
Sep 16 19:22:29.979584 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:29.979251    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerDied","Data":"92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503"}
Sep 16 19:22:30.985919 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:30.985884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerStarted","Data":"1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100"}
Sep 16 19:22:30.986364 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:30.985975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerStarted","Data":"0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2"}
Sep 16 19:22:30.986364 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:30.986200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:31.009572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:31.009519    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" podStartSLOduration=3.433678332 podStartE2EDuration="7.00950302s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.575824688s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:31.007629056 +0000 UTC m=+2353.424331818" watchObservedRunningTime="2026-09-16 19:22:31.00950302 +0000 UTC m=+2353.426205779"
Sep 16 19:22:31.990037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:31.990006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:33.689645 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:33.689605    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:22:38.000651 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:38.000621    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:22:43.690251 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:43.690217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:22:49.869873 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:49.869834    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:49.894735 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:49.894692    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"]
Sep 16 19:22:49.894948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:49.894917    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" containerID="cri-o://b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5" gracePeriod=30
Sep 16 19:22:49.895111 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:49.894961    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kube-rbac-proxy" containerID="cri-o://0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2" gracePeriod=30
Sep 16 19:22:49.932937 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:49.932905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:50.072246 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:50.072212    2562 generic.go:356] "Generic (PLEG): container finished" podID="da51a222-ff7d-462e-86b0-bf74060d4363" containerID="0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2" exitCode=2
Sep 16 19:22:50.072436 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:50.072293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerDied","Data":"0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2"}
Sep 16 19:22:53.684031 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:53.683989    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 16 19:22:53.689277 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:53.689253    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 16 19:22:54.935919 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.935889    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:22:54.983687 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.983613    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9f72\" (UniqueName: \"kubernetes.io/projected/da51a222-ff7d-462e-86b0-bf74060d4363-kube-api-access-h9f72\") pod \"da51a222-ff7d-462e-86b0-bf74060d4363\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") "
Sep 16 19:22:54.983687 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.983680    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-cabundle-cert\") pod \"da51a222-ff7d-462e-86b0-bf74060d4363\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") "
Sep 16 19:22:54.983905 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.983702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-kube-rbac-proxy-sar-config\") pod \"da51a222-ff7d-462e-86b0-bf74060d4363\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") "
Sep 16 19:22:54.983905 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.983731    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da51a222-ff7d-462e-86b0-bf74060d4363-kserve-provision-location\") pod \"da51a222-ff7d-462e-86b0-bf74060d4363\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") "
Sep 16 19:22:54.983905 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.983789    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls\") pod \"da51a222-ff7d-462e-86b0-bf74060d4363\" (UID: \"da51a222-ff7d-462e-86b0-bf74060d4363\") "
Sep 16 19:22:54.984113 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.984085    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da51a222-ff7d-462e-86b0-bf74060d4363-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da51a222-ff7d-462e-86b0-bf74060d4363" (UID: "da51a222-ff7d-462e-86b0-bf74060d4363"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:54.984113 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.984098    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da51a222-ff7d-462e-86b0-bf74060d4363" (UID: "da51a222-ff7d-462e-86b0-bf74060d4363"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:54.984205 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.984090    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da51a222-ff7d-462e-86b0-bf74060d4363" (UID: "da51a222-ff7d-462e-86b0-bf74060d4363"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:54.985789 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.985768    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da51a222-ff7d-462e-86b0-bf74060d4363-kube-api-access-h9f72" (OuterVolumeSpecName: "kube-api-access-h9f72") pod "da51a222-ff7d-462e-86b0-bf74060d4363" (UID: "da51a222-ff7d-462e-86b0-bf74060d4363"). InnerVolumeSpecName "kube-api-access-h9f72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:54.986017 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:54.985995    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da51a222-ff7d-462e-86b0-bf74060d4363" (UID: "da51a222-ff7d-462e-86b0-bf74060d4363"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:55.084527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.084493    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.084527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.084521    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da51a222-ff7d-462e-86b0-bf74060d4363-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.084527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.084531    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da51a222-ff7d-462e-86b0-bf74060d4363-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.084754 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.084540    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9f72\" (UniqueName: \"kubernetes.io/projected/da51a222-ff7d-462e-86b0-bf74060d4363-kube-api-access-h9f72\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.084754 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.084549    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da51a222-ff7d-462e-86b0-bf74060d4363-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:22:55.095927 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.095899    2562 generic.go:356] "Generic (PLEG): container finished" podID="da51a222-ff7d-462e-86b0-bf74060d4363" containerID="b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5" exitCode=0
Sep 16 19:22:55.096098 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.095977    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"
Sep 16 19:22:55.096098 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.095979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerDied","Data":"b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5"}
Sep 16 19:22:55.096098 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.096074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd" event={"ID":"da51a222-ff7d-462e-86b0-bf74060d4363","Type":"ContainerDied","Data":"778c653e60c97f192d9049fa8034436ab16c3e5d4c4557e2607a6086f822fd74"}
Sep 16 19:22:55.096266 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.096122    2562 scope.go:117] "RemoveContainer" containerID="0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2"
Sep 16 19:22:55.105934 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.105920    2562 scope.go:117] "RemoveContainer" containerID="b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5"
Sep 16 19:22:55.113707 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.113679    2562 scope.go:117] "RemoveContainer" containerID="b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0"
Sep 16 19:22:55.121381 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.121371    2562 scope.go:117] "RemoveContainer" containerID="0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2"
Sep 16 19:22:55.121683 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:55.121658    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2\": container with ID starting with 0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2 not found: ID does not exist" containerID="0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2"
Sep 16 19:22:55.121752 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.121691    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2"} err="failed to get container status \"0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2\": rpc error: code = NotFound desc = could not find container \"0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2\": container with ID starting with 0ea8f9c3c7d83ec0e61928541ede542b4841c4145a6870c225ee247b75dfa3a2 not found: ID does not exist"
Sep 16 19:22:55.121752 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.121713    2562 scope.go:117] "RemoveContainer" containerID="b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5"
Sep 16 19:22:55.121940 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:55.121922    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5\": container with ID starting with b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5 not found: ID does not exist" containerID="b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5"
Sep 16 19:22:55.121997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.121948    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5"} err="failed to get container status \"b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5\": rpc error: code = NotFound desc = could not find container \"b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5\": container with ID starting with b6c54d6aceaf6ef01c9b084e470b5875ef83f8c6b271293df3561166fbc9cce5 not found: ID does not exist"
Sep 16 19:22:55.121997 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.121964    2562 scope.go:117] "RemoveContainer" containerID="b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0"
Sep 16 19:22:55.122185 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:22:55.122168    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0\": container with ID starting with b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0 not found: ID does not exist" containerID="b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0"
Sep 16 19:22:55.122250 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.122189    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0"} err="failed to get container status \"b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0\": rpc error: code = NotFound desc = could not find container \"b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0\": container with ID starting with b7af47322854889f94a301f4c190ddd1bd3e640911d7036e8e0ac9a6d45fd1d0 not found: ID does not exist"
Sep 16 19:22:55.123257 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.123236    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"]
Sep 16 19:22:55.125144 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:55.125125    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7bb9c548b5-xr4pd"]
Sep 16 19:22:56.121894 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:22:56.121862    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" path="/var/lib/kubelet/pods/da51a222-ff7d-462e-86b0-bf74060d4363/volumes"
Sep 16 19:23:00.114382 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:00.114348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:00.377734 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:00.377636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:08.004799 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:08.004769    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:23:14.312284 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:14.312253    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"]
Sep 16 19:23:14.312688 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:14.312506    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kserve-container" containerID="cri-o://0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2" gracePeriod=30
Sep 16 19:23:14.312688 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:14.312549    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kube-rbac-proxy" containerID="cri-o://1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100" gracePeriod=30
Sep 16 19:23:14.340613 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:14.340584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:15.184234 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:15.184200    2562 generic.go:356] "Generic (PLEG): container finished" podID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerID="1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100" exitCode=2
Sep 16 19:23:15.184234 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:15.184237    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerDied","Data":"1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100"}
Sep 16 19:23:17.994936 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:17.994894    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 16 19:23:18.001418 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:18.001382    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 16 19:23:19.452372 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:19.452335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:19.886947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:19.886902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:20.549568 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.549543    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:23:20.709814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.709730    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls\") pod \"90619afa-84ae-4550-abd4-6cdbeddbce49\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") "
Sep 16 19:23:20.709814 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.709774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-rbac-proxy-sar-config\") pod \"90619afa-84ae-4550-abd4-6cdbeddbce49\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") "
Sep 16 19:23:20.710045 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.709836    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90619afa-84ae-4550-abd4-6cdbeddbce49-kserve-provision-location\") pod \"90619afa-84ae-4550-abd4-6cdbeddbce49\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") "
Sep 16 19:23:20.710045 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.709858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-cabundle-cert\") pod \"90619afa-84ae-4550-abd4-6cdbeddbce49\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") "
Sep 16 19:23:20.710045 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.709911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8dsb\" (UniqueName: \"kubernetes.io/projected/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-api-access-c8dsb\") pod \"90619afa-84ae-4550-abd4-6cdbeddbce49\" (UID: \"90619afa-84ae-4550-abd4-6cdbeddbce49\") "
Sep 16 19:23:20.710238 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.710214    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90619afa-84ae-4550-abd4-6cdbeddbce49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90619afa-84ae-4550-abd4-6cdbeddbce49" (UID: "90619afa-84ae-4550-abd4-6cdbeddbce49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:20.710296 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.710219    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90619afa-84ae-4550-abd4-6cdbeddbce49" (UID: "90619afa-84ae-4550-abd4-6cdbeddbce49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:20.710374 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.710309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90619afa-84ae-4550-abd4-6cdbeddbce49" (UID: "90619afa-84ae-4550-abd4-6cdbeddbce49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:20.712055 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.712033    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90619afa-84ae-4550-abd4-6cdbeddbce49" (UID: "90619afa-84ae-4550-abd4-6cdbeddbce49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:20.712163 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.712141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-api-access-c8dsb" (OuterVolumeSpecName: "kube-api-access-c8dsb") pod "90619afa-84ae-4550-abd4-6cdbeddbce49" (UID: "90619afa-84ae-4550-abd4-6cdbeddbce49"). InnerVolumeSpecName "kube-api-access-c8dsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:20.811079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.811050    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90619afa-84ae-4550-abd4-6cdbeddbce49-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:23:20.811079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.811075    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:23:20.811079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.811085    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90619afa-84ae-4550-abd4-6cdbeddbce49-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:23:20.811275 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.811094    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90619afa-84ae-4550-abd4-6cdbeddbce49-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:23:20.811275 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:20.811104    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8dsb\" (UniqueName: \"kubernetes.io/projected/90619afa-84ae-4550-abd4-6cdbeddbce49-kube-api-access-c8dsb\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:23:21.209558 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.209522    2562 generic.go:356] "Generic (PLEG): container finished" podID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerID="0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2" exitCode=0
Sep 16 19:23:21.209723 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.209589    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"
Sep 16 19:23:21.209723 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.209591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerDied","Data":"0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2"}
Sep 16 19:23:21.209723 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.209653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm" event={"ID":"90619afa-84ae-4550-abd4-6cdbeddbce49","Type":"ContainerDied","Data":"f264e80a5a36e92418208dce363d40a0343f28d25a1f2152ce9e5afdf36f0cae"}
Sep 16 19:23:21.209723 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.209690    2562 scope.go:117] "RemoveContainer" containerID="1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100"
Sep 16 19:23:21.218567 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.218547    2562 scope.go:117] "RemoveContainer" containerID="0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2"
Sep 16 19:23:21.228465 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.228441    2562 scope.go:117] "RemoveContainer" containerID="92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503"
Sep 16 19:23:21.235979 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.235955    2562 scope.go:117] "RemoveContainer" containerID="1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100"
Sep 16 19:23:21.236331 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:23:21.236294    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100\": container with ID starting with 1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100 not found: ID does not exist" containerID="1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100"
Sep 16 19:23:21.236397 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.236355    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100"} err="failed to get container status \"1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100\": rpc error: code = NotFound desc = could not find container \"1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100\": container with ID starting with 1071a49aea5efa83266c8a0f239427fb2fca0bf6f4881f1bd0b049521480b100 not found: ID does not exist"
Sep 16 19:23:21.236397 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.236382    2562 scope.go:117] "RemoveContainer" containerID="0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2"
Sep 16 19:23:21.236694 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:23:21.236661    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2\": container with ID starting with 0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2 not found: ID does not exist" containerID="0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2"
Sep 16 19:23:21.236831 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.236706    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2"} err="failed to get container status \"0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2\": rpc error: code = NotFound desc = could not find container \"0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2\": container with ID starting with 0223e9afad62f22939d3f7463ddd0c9b8c3777dcb0fffb53680134375c9404f2 not found: ID does not exist"
Sep 16 19:23:21.236831 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.236738    2562 scope.go:117] "RemoveContainer" containerID="92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503"
Sep 16 19:23:21.237262 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:23:21.237228    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503\": container with ID starting with 92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503 not found: ID does not exist" containerID="92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503"
Sep 16 19:23:21.237380 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.237267    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503"} err="failed to get container status \"92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503\": rpc error: code = NotFound desc = could not find container \"92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503\": container with ID starting with 92e500298f5d2db50cbdfefff321a2dcfdc32dde39a0441dae546e713c6c6503 not found: ID does not exist"
Sep 16 19:23:21.239620 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.239598    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"]
Sep 16 19:23:21.243191 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:21.243170    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d5prm"]
Sep 16 19:23:22.122166 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:22.122136    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" path="/var/lib/kubelet/pods/90619afa-84ae-4550-abd4-6cdbeddbce49/volumes"
Sep 16 19:23:51.920778 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:23:51.920736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.239718 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:30.239679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:35.364966 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.364935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:35.407420 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407386    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"]
Sep 16 19:24:35.407746 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407729    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="storage-initializer"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407749    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="storage-initializer"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407779    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kube-rbac-proxy"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407788    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kube-rbac-proxy"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407797    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kserve-container"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407806    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kserve-container"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407816    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kube-rbac-proxy"
Sep 16 19:24:35.407825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407824    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kube-rbac-proxy"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407841    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="storage-initializer"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407849    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="storage-initializer"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407864    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="storage-initializer"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407873    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="storage-initializer"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407883    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407890    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407898    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kube-rbac-proxy"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407907    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kube-rbac-proxy"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407919    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kserve-container"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407928    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kserve-container"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.407999    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kube-rbac-proxy"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.408014    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kube-rbac-proxy"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.408024    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kube-rbac-proxy"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.408034    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="90619afa-84ae-4550-abd4-6cdbeddbce49" containerName="kserve-container"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.408043    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bb16bc-7304-4345-82c1-182ce5651130" containerName="kserve-container"
Sep 16 19:24:35.408176 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.408055    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da51a222-ff7d-462e-86b0-bf74060d4363" containerName="kserve-container"
Sep 16 19:24:35.410960 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.410943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.413357 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.413332    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:24:35.413357 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.413343    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 16 19:24:35.413550 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.413358    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:24:35.413671 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.413650    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:24:35.413749 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.413699    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:24:35.414231 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.414211    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:24:35.414360 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.414213    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:24:35.420088 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.420068    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"]
Sep 16 19:24:35.544486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.544451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ad20e2-bc46-4d3c-8052-35cec08892e5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.544486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.544488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.544709 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.544511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.544709 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.544608    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.544709 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.544668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq296\" (UniqueName: \"kubernetes.io/projected/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-api-access-zq296\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645231 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645231 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq296\" (UniqueName: \"kubernetes.io/projected/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-api-access-zq296\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ad20e2-bc46-4d3c-8052-35cec08892e5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645558 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645615 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:24:35.645567    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 16 19:24:35.645676 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:24:35.645646    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls podName:08ad20e2-bc46-4d3c-8052-35cec08892e5 nodeName:}" failed. No retries permitted until 2026-09-16 19:24:36.14562605 +0000 UTC m=+2478.562328790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" (UID: "08ad20e2-bc46-4d3c-8052-35cec08892e5") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 16 19:24:35.645745 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ad20e2-bc46-4d3c-8052-35cec08892e5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.645880 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.646005 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.645988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.654349 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.654310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq296\" (UniqueName: \"kubernetes.io/projected/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-api-access-zq296\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:35.892045 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:35.892009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:36.150392 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:36.150349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:36.152890 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:36.152867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-888mm\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:36.322102 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:36.322067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:36.592365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:36.592335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"]
Sep 16 19:24:36.594616 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:24:36.594591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ad20e2_bc46_4d3c_8052_35cec08892e5.slice/crio-55512177c76ff6e491c709cd58e46f5664c056e73600a46f92c70102d4733508 WatchSource:0}: Error finding container 55512177c76ff6e491c709cd58e46f5664c056e73600a46f92c70102d4733508: Status 404 returned error can't find the container with id 55512177c76ff6e491c709cd58e46f5664c056e73600a46f92c70102d4733508
Sep 16 19:24:37.527246 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:37.527209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerStarted","Data":"8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c"}
Sep 16 19:24:37.527466 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:37.527283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerStarted","Data":"55512177c76ff6e491c709cd58e46f5664c056e73600a46f92c70102d4733508"}
Sep 16 19:24:39.581028 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:39.580933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:40.542880 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:40.542836    2562 generic.go:356] "Generic (PLEG): container finished" podID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerID="8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c" exitCode=0
Sep 16 19:24:40.543012 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:40.542909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerDied","Data":"8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c"}
Sep 16 19:24:40.544184 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:40.544168    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:24:44.727041 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.727002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:44.763478 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.763444    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"]
Sep 16 19:24:44.790293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.790260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"]
Sep 16 19:24:44.790479 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.790416    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:44.793004 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.792972    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:24:44.793004 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.792971    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 16 19:24:44.813124 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.813097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:44.932036 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.931986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc9ace5-87e5-4d0d-96af-a84b73680000-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:44.932238 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.932050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtlh\" (UniqueName: \"kubernetes.io/projected/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-api-access-sqtlh\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:44.932238 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.932088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:44.932238 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.932207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:44.932426 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:44.932267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc9ace5-87e5-4d0d-96af-a84b73680000-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.033497 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.033400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.033497 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.033487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc9ace5-87e5-4d0d-96af-a84b73680000-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.033755 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.033546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc9ace5-87e5-4d0d-96af-a84b73680000-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.033755 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.033578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtlh\" (UniqueName: \"kubernetes.io/projected/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-api-access-sqtlh\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.033755 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.033609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.034144 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.034118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc9ace5-87e5-4d0d-96af-a84b73680000-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.034385 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.034364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.034455 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.034398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.036778 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.036750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc9ace5-87e5-4d0d-96af-a84b73680000-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.042803 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.042759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtlh\" (UniqueName: \"kubernetes.io/projected/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-api-access-sqtlh\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:45.102656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:45.102620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:46.891985 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:46.891958    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"]
Sep 16 19:24:46.893666 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:24:46.893640    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc9ace5_87e5_4d0d_96af_a84b73680000.slice/crio-224c89a189d935253852832f770fdbe1d16215ad8a91c861dc02505c0302d154 WatchSource:0}: Error finding container 224c89a189d935253852832f770fdbe1d16215ad8a91c861dc02505c0302d154: Status 404 returned error can't find the container with id 224c89a189d935253852832f770fdbe1d16215ad8a91c861dc02505c0302d154
Sep 16 19:24:47.577679 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.577643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerStarted","Data":"a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056"}
Sep 16 19:24:47.577890 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.577743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerStarted","Data":"224c89a189d935253852832f770fdbe1d16215ad8a91c861dc02505c0302d154"}
Sep 16 19:24:47.579799 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.579774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerStarted","Data":"4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0"}
Sep 16 19:24:47.579903 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.579826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerStarted","Data":"19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229"}
Sep 16 19:24:47.580157 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.580138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:47.580200 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.580189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:47.581420 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.581397    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:24:47.614978 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:47.614925    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podStartSLOduration=2.629697515 podStartE2EDuration="12.614907382s" totalImagesPullingTime="6.158456989s" totalInitContainerRuntime="3.826752878s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:47.612904696 +0000 UTC m=+2490.029607457" watchObservedRunningTime="2026-09-16 19:24:47.614907382 +0000 UTC m=+2490.031610140"
Sep 16 19:24:48.585035 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:48.584983    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:24:51.599650 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:51.599610    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerID="a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056" exitCode=0
Sep 16 19:24:51.600119 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:51.599680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerDied","Data":"a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056"}
Sep 16 19:24:52.606534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:52.606499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerStarted","Data":"95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718"}
Sep 16 19:24:52.606877 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:52.606558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerStarted","Data":"cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b"}
Sep 16 19:24:52.606877 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:52.606756    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:52.606877 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:52.606799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:24:52.641310 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:52.641261    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" podStartSLOduration=4.624946821 podStartE2EDuration="8.641247488s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.016300667s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:52.638773982 +0000 UTC m=+2495.055476741" watchObservedRunningTime="2026-09-16 19:24:52.641247488 +0000 UTC m=+2495.057950247"
Sep 16 19:24:53.589180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:53.589152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:24:53.589661 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:53.589636    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:24:58.616841 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:24:58.616814    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:25:03.590085 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:03.590034    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:25:13.590341 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:13.590235    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:25:23.590312 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:23.590269    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:25:28.697200 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:28.697138    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:25:33.590353 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:33.590292    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:25:38.619766 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:38.619736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:25:43.589810 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:43.589773    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:25:44.828610 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:44.828571    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"]
Sep 16 19:25:44.829071 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:44.828836    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kserve-container" containerID="cri-o://cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b" gracePeriod=30
Sep 16 19:25:44.829071 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:44.828891    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kube-rbac-proxy" containerID="cri-o://95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718" gracePeriod=30
Sep 16 19:25:44.861626 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:44.861596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:45.847220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:45.847184    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerID="95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718" exitCode=2
Sep 16 19:25:45.847609 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:45.847255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerDied","Data":"95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718"}
Sep 16 19:25:48.611622 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:48.611580    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 16 19:25:48.618092 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:48.618050    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.55:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 16 19:25:50.099781 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:50.099740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:50.758013 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:50.757971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:52.272412 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.272390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:25:52.316777 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.316752    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc9ace5-87e5-4d0d-96af-a84b73680000-proxy-tls\") pod \"ebc9ace5-87e5-4d0d-96af-a84b73680000\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") "
Sep 16 19:25:52.316951 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.316784    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqtlh\" (UniqueName: \"kubernetes.io/projected/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-api-access-sqtlh\") pod \"ebc9ace5-87e5-4d0d-96af-a84b73680000\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") "
Sep 16 19:25:52.316951 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.316813    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc9ace5-87e5-4d0d-96af-a84b73680000-kserve-provision-location\") pod \"ebc9ace5-87e5-4d0d-96af-a84b73680000\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") "
Sep 16 19:25:52.317087 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.316947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-rbac-proxy-sar-config\") pod \"ebc9ace5-87e5-4d0d-96af-a84b73680000\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") "
Sep 16 19:25:52.317087 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.316981    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-cabundle-cert\") pod \"ebc9ace5-87e5-4d0d-96af-a84b73680000\" (UID: \"ebc9ace5-87e5-4d0d-96af-a84b73680000\") "
Sep 16 19:25:52.317221 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.317117    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc9ace5-87e5-4d0d-96af-a84b73680000-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebc9ace5-87e5-4d0d-96af-a84b73680000" (UID: "ebc9ace5-87e5-4d0d-96af-a84b73680000"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:52.317290 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.317265    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebc9ace5-87e5-4d0d-96af-a84b73680000" (UID: "ebc9ace5-87e5-4d0d-96af-a84b73680000"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:52.317369 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.317276    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc9ace5-87e5-4d0d-96af-a84b73680000-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:25:52.317577 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.317473    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebc9ace5-87e5-4d0d-96af-a84b73680000" (UID: "ebc9ace5-87e5-4d0d-96af-a84b73680000"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:52.319119 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.319093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc9ace5-87e5-4d0d-96af-a84b73680000-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebc9ace5-87e5-4d0d-96af-a84b73680000" (UID: "ebc9ace5-87e5-4d0d-96af-a84b73680000"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:52.319222 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.319130    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-api-access-sqtlh" (OuterVolumeSpecName: "kube-api-access-sqtlh") pod "ebc9ace5-87e5-4d0d-96af-a84b73680000" (UID: "ebc9ace5-87e5-4d0d-96af-a84b73680000"). InnerVolumeSpecName "kube-api-access-sqtlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:52.418642 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.418571    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:25:52.418642 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.418595    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc9ace5-87e5-4d0d-96af-a84b73680000-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:25:52.418642 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.418604    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqtlh\" (UniqueName: \"kubernetes.io/projected/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-api-access-sqtlh\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:25:52.418642 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.418615    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc9ace5-87e5-4d0d-96af-a84b73680000-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:25:52.879065 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.879031    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerID="cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b" exitCode=0
Sep 16 19:25:52.879233 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.879082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerDied","Data":"cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b"}
Sep 16 19:25:52.879233 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.879089    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"
Sep 16 19:25:52.879233 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.879129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b" event={"ID":"ebc9ace5-87e5-4d0d-96af-a84b73680000","Type":"ContainerDied","Data":"224c89a189d935253852832f770fdbe1d16215ad8a91c861dc02505c0302d154"}
Sep 16 19:25:52.879233 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.879155    2562 scope.go:117] "RemoveContainer" containerID="95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718"
Sep 16 19:25:52.888812 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.888794    2562 scope.go:117] "RemoveContainer" containerID="cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b"
Sep 16 19:25:52.897656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.897632    2562 scope.go:117] "RemoveContainer" containerID="a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056"
Sep 16 19:25:52.901166 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.901143    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"]
Sep 16 19:25:52.906246 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.906219    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-jgc5b"]
Sep 16 19:25:52.906897 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.906879    2562 scope.go:117] "RemoveContainer" containerID="95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718"
Sep 16 19:25:52.907204 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:25:52.907188    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718\": container with ID starting with 95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718 not found: ID does not exist" containerID="95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718"
Sep 16 19:25:52.907253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.907211    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718"} err="failed to get container status \"95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718\": rpc error: code = NotFound desc = could not find container \"95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718\": container with ID starting with 95e9ecee467025307b39cb0716b07fd5b4dbe1e8f8e98cbefcce5186a97f6718 not found: ID does not exist"
Sep 16 19:25:52.907253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.907227    2562 scope.go:117] "RemoveContainer" containerID="cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b"
Sep 16 19:25:52.907505 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:25:52.907488    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b\": container with ID starting with cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b not found: ID does not exist" containerID="cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b"
Sep 16 19:25:52.907555 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.907512    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b"} err="failed to get container status \"cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b\": rpc error: code = NotFound desc = could not find container \"cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b\": container with ID starting with cf24ca8b09e0d34928e4da45ef67825f8b4ba6eee7e73a38cd00d3ca77770c6b not found: ID does not exist"
Sep 16 19:25:52.907555 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.907527    2562 scope.go:117] "RemoveContainer" containerID="a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056"
Sep 16 19:25:52.907811 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:25:52.907794    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056\": container with ID starting with a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056 not found: ID does not exist" containerID="a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056"
Sep 16 19:25:52.907848 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:52.907818    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056"} err="failed to get container status \"a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056\": rpc error: code = NotFound desc = could not find container \"a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056\": container with ID starting with a8fdb39616338a2e47ce4da5a9eb9bb6b26b9be3e1e156e5475b19ed8f8e8056 not found: ID does not exist"
Sep 16 19:25:53.589933 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:53.589887    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 16 19:25:54.122231 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:25:54.122200    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" path="/var/lib/kubelet/pods/ebc9ace5-87e5-4d0d-96af-a84b73680000/volumes"
Sep 16 19:26:03.591142 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:03.591114    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:26:06.033981 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:06.033951    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"]
Sep 16 19:26:06.034435 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:06.034212    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container" containerID="cri-o://19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229" gracePeriod=30
Sep 16 19:26:06.034435 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:06.034267    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kube-rbac-proxy" containerID="cri-o://4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0" gracePeriod=30
Sep 16 19:26:06.045885 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:06.045857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:06.950456 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:06.950418    2562 generic.go:356] "Generic (PLEG): container finished" podID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerID="4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0" exitCode=2
Sep 16 19:26:06.950636 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:06.950464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerDied","Data":"4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0"}
Sep 16 19:26:08.585948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:08.585909    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 16 19:26:11.177522 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.177489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:11.202154 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202122    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"]
Sep 16 19:26:11.202669 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202651    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kserve-container"
Sep 16 19:26:11.202725 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202675    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kserve-container"
Sep 16 19:26:11.202725 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202694    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="storage-initializer"
Sep 16 19:26:11.202725 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202704    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="storage-initializer"
Sep 16 19:26:11.202823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202734    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kube-rbac-proxy"
Sep 16 19:26:11.202823 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202743    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kube-rbac-proxy"
Sep 16 19:26:11.202885 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202830    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kserve-container"
Sep 16 19:26:11.202885 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.202847    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc9ace5-87e5-4d0d-96af-a84b73680000" containerName="kube-rbac-proxy"
Sep 16 19:26:11.206466 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.206445    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.208729 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.208712    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 16 19:26:11.209572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.209531    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:26:11.216473 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.216435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"]
Sep 16 19:26:11.277299 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.277278    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8jd\" (UniqueName: \"kubernetes.io/projected/40378a57-440a-4395-98b7-c06dec99dbdb-kube-api-access-zz8jd\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.277427 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.277359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.277427 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.277387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.277427 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.277403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40378a57-440a-4395-98b7-c06dec99dbdb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.277550 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.277442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.280807 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.280786    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:26:11.378792 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ad20e2-bc46-4d3c-8052-35cec08892e5-kserve-provision-location\") pod \"08ad20e2-bc46-4d3c-8052-35cec08892e5\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") "
Sep 16 19:26:11.378792 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378759    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-cabundle-cert\") pod \"08ad20e2-bc46-4d3c-8052-35cec08892e5\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") "
Sep 16 19:26:11.378792 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378779    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq296\" (UniqueName: \"kubernetes.io/projected/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-api-access-zq296\") pod \"08ad20e2-bc46-4d3c-8052-35cec08892e5\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") "
Sep 16 19:26:11.379037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378819    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls\") pod \"08ad20e2-bc46-4d3c-8052-35cec08892e5\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") "
Sep 16 19:26:11.379037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-rbac-proxy-sar-config\") pod \"08ad20e2-bc46-4d3c-8052-35cec08892e5\" (UID: \"08ad20e2-bc46-4d3c-8052-35cec08892e5\") "
Sep 16 19:26:11.379037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378958    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.379037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.378995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.379037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40378a57-440a-4395-98b7-c06dec99dbdb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.379283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.379283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379075    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ad20e2-bc46-4d3c-8052-35cec08892e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08ad20e2-bc46-4d3c-8052-35cec08892e5" (UID: "08ad20e2-bc46-4d3c-8052-35cec08892e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:11.379283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8jd\" (UniqueName: \"kubernetes.io/projected/40378a57-440a-4395-98b7-c06dec99dbdb-kube-api-access-zz8jd\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.379283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379090    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08ad20e2-bc46-4d3c-8052-35cec08892e5" (UID: "08ad20e2-bc46-4d3c-8052-35cec08892e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:11.379283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379257    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "08ad20e2-bc46-4d3c-8052-35cec08892e5" (UID: "08ad20e2-bc46-4d3c-8052-35cec08892e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:11.379283 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379284    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ad20e2-bc46-4d3c-8052-35cec08892e5-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:26:11.379718 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:26:11.379294    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:26:11.379718 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379302    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:26:11.379718 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:26:11.379385    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls podName:40378a57-440a-4395-98b7-c06dec99dbdb nodeName:}" failed. No retries permitted until 2026-09-16 19:26:11.879364487 +0000 UTC m=+2574.296067230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" (UID: "40378a57-440a-4395-98b7-c06dec99dbdb") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:26:11.379718 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40378a57-440a-4395-98b7-c06dec99dbdb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.379956 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.379814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.380089 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.380068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.381122 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.381104    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08ad20e2-bc46-4d3c-8052-35cec08892e5" (UID: "08ad20e2-bc46-4d3c-8052-35cec08892e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:11.381221 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.381201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-api-access-zq296" (OuterVolumeSpecName: "kube-api-access-zq296") pod "08ad20e2-bc46-4d3c-8052-35cec08892e5" (UID: "08ad20e2-bc46-4d3c-8052-35cec08892e5"). InnerVolumeSpecName "kube-api-access-zq296". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:11.391622 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.391602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8jd\" (UniqueName: \"kubernetes.io/projected/40378a57-440a-4395-98b7-c06dec99dbdb-kube-api-access-zz8jd\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.480463 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.480433    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq296\" (UniqueName: \"kubernetes.io/projected/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-api-access-zq296\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:26:11.480463 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.480459    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08ad20e2-bc46-4d3c-8052-35cec08892e5-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:26:11.480463 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.480471    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08ad20e2-bc46-4d3c-8052-35cec08892e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:26:11.486893 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.486857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:11.884953 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.884906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.887465 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.887447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:11.979895 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.979864    2562 generic.go:356] "Generic (PLEG): container finished" podID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerID="19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229" exitCode=0
Sep 16 19:26:11.980069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.979920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerDied","Data":"19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229"}
Sep 16 19:26:11.980069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.979987    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm" event={"ID":"08ad20e2-bc46-4d3c-8052-35cec08892e5","Type":"ContainerDied","Data":"55512177c76ff6e491c709cd58e46f5664c056e73600a46f92c70102d4733508"}
Sep 16 19:26:11.980069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.979920    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"
Sep 16 19:26:11.980069 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.980012    2562 scope.go:117] "RemoveContainer" containerID="4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0"
Sep 16 19:26:11.988942 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.988926    2562 scope.go:117] "RemoveContainer" containerID="19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229"
Sep 16 19:26:11.996238 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:11.996222    2562 scope.go:117] "RemoveContainer" containerID="8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c"
Sep 16 19:26:12.001703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.001680    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"]
Sep 16 19:26:12.004986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.004968    2562 scope.go:117] "RemoveContainer" containerID="4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0"
Sep 16 19:26:12.005273 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:26:12.005248    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0\": container with ID starting with 4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0 not found: ID does not exist" containerID="4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0"
Sep 16 19:26:12.005387 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.005283    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0"} err="failed to get container status \"4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0\": rpc error: code = NotFound desc = could not find container \"4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0\": container with ID starting with 4c6bf4e178ca11dd93b0c3ad79247f72e73e2197ce43946e5c56989e615c38c0 not found: ID does not exist"
Sep 16 19:26:12.005387 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.005307    2562 scope.go:117] "RemoveContainer" containerID="19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229"
Sep 16 19:26:12.005621 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:26:12.005604    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229\": container with ID starting with 19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229 not found: ID does not exist" containerID="19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229"
Sep 16 19:26:12.005673 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.005626    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229"} err="failed to get container status \"19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229\": rpc error: code = NotFound desc = could not find container \"19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229\": container with ID starting with 19ac4669972eff14a2b3d6ac617eb40c266f87954d81bfb246f7b17aef6cc229 not found: ID does not exist"
Sep 16 19:26:12.005673 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.005641    2562 scope.go:117] "RemoveContainer" containerID="8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c"
Sep 16 19:26:12.005746 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.005669    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-888mm"]
Sep 16 19:26:12.005904 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:26:12.005889    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c\": container with ID starting with 8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c not found: ID does not exist" containerID="8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c"
Sep 16 19:26:12.005952 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.005910    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c"} err="failed to get container status \"8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c\": rpc error: code = NotFound desc = could not find container \"8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c\": container with ID starting with 8b960dd514cc5717b87ba011ce3d5f24cc8e109c3f45b625cfc78756bf1e050c not found: ID does not exist"
Sep 16 19:26:12.117618 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.117586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:12.122174 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.122155    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" path="/var/lib/kubelet/pods/08ad20e2-bc46-4d3c-8052-35cec08892e5/volumes"
Sep 16 19:26:12.391051 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.391008    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"]
Sep 16 19:26:12.393411 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:26:12.393376    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40378a57_440a_4395_98b7_c06dec99dbdb.slice/crio-1f29d04146c3775466c215f984649cee7e9aeb4747514cbf9cf6edd480a1d6c1 WatchSource:0}: Error finding container 1f29d04146c3775466c215f984649cee7e9aeb4747514cbf9cf6edd480a1d6c1: Status 404 returned error can't find the container with id 1f29d04146c3775466c215f984649cee7e9aeb4747514cbf9cf6edd480a1d6c1
Sep 16 19:26:12.987043 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.987011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerStarted","Data":"20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c"}
Sep 16 19:26:12.987257 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:12.987078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerStarted","Data":"1f29d04146c3775466c215f984649cee7e9aeb4747514cbf9cf6edd480a1d6c1"}
Sep 16 19:26:17.007264 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:17.007228    2562 generic.go:356] "Generic (PLEG): container finished" podID="40378a57-440a-4395-98b7-c06dec99dbdb" containerID="20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c" exitCode=0
Sep 16 19:26:17.007759 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:17.007272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerDied","Data":"20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c"}
Sep 16 19:26:18.015246 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:18.015206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerStarted","Data":"226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf"}
Sep 16 19:26:18.015655 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:18.015314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerStarted","Data":"4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70"}
Sep 16 19:26:18.015655 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:18.015559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:18.015760 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:18.015700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:18.035463 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:18.035419    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" podStartSLOduration=3.195700139 podStartE2EDuration="7.035395917s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.839695778s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:18.03387005 +0000 UTC m=+2580.450572823" watchObservedRunningTime="2026-09-16 19:26:18.035395917 +0000 UTC m=+2580.452098675"
Sep 16 19:26:24.028751 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:24.028722    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:26:54.032248 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:26:54.032218    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:27:01.321925 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:01.321887    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"]
Sep 16 19:27:01.322549 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:01.322215    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kserve-container" containerID="cri-o://4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70" gracePeriod=30
Sep 16 19:27:01.322549 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:01.322230    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kube-rbac-proxy" containerID="cri-o://226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf" gracePeriod=30
Sep 16 19:27:01.351959 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:01.351925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:02.216865 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.216834    2562 generic.go:356] "Generic (PLEG): container finished" podID="40378a57-440a-4395-98b7-c06dec99dbdb" containerID="226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf" exitCode=2
Sep 16 19:27:02.216983 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.216896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerDied","Data":"226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf"}
Sep 16 19:27:02.368931 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.368909    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:27:02.505904 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.505825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40378a57-440a-4395-98b7-c06dec99dbdb-kserve-provision-location\") pod \"40378a57-440a-4395-98b7-c06dec99dbdb\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") "
Sep 16 19:27:02.505904 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.505879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-cabundle-cert\") pod \"40378a57-440a-4395-98b7-c06dec99dbdb\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") "
Sep 16 19:27:02.506106 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.505916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz8jd\" (UniqueName: \"kubernetes.io/projected/40378a57-440a-4395-98b7-c06dec99dbdb-kube-api-access-zz8jd\") pod \"40378a57-440a-4395-98b7-c06dec99dbdb\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") "
Sep 16 19:27:02.506106 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.505948    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-kube-rbac-proxy-sar-config\") pod \"40378a57-440a-4395-98b7-c06dec99dbdb\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") "
Sep 16 19:27:02.506106 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.505997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls\") pod \"40378a57-440a-4395-98b7-c06dec99dbdb\" (UID: \"40378a57-440a-4395-98b7-c06dec99dbdb\") "
Sep 16 19:27:02.506274 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.506256    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40378a57-440a-4395-98b7-c06dec99dbdb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40378a57-440a-4395-98b7-c06dec99dbdb" (UID: "40378a57-440a-4395-98b7-c06dec99dbdb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:02.506354 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.506310    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40378a57-440a-4395-98b7-c06dec99dbdb" (UID: "40378a57-440a-4395-98b7-c06dec99dbdb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:02.506406 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.506358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40378a57-440a-4395-98b7-c06dec99dbdb" (UID: "40378a57-440a-4395-98b7-c06dec99dbdb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:02.508142 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.508122    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40378a57-440a-4395-98b7-c06dec99dbdb" (UID: "40378a57-440a-4395-98b7-c06dec99dbdb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:02.508259 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.508241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40378a57-440a-4395-98b7-c06dec99dbdb-kube-api-access-zz8jd" (OuterVolumeSpecName: "kube-api-access-zz8jd") pod "40378a57-440a-4395-98b7-c06dec99dbdb" (UID: "40378a57-440a-4395-98b7-c06dec99dbdb"). InnerVolumeSpecName "kube-api-access-zz8jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:02.606947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.606913    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40378a57-440a-4395-98b7-c06dec99dbdb-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:27:02.606947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.606940    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:27:02.606947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.606953    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz8jd\" (UniqueName: \"kubernetes.io/projected/40378a57-440a-4395-98b7-c06dec99dbdb-kube-api-access-zz8jd\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:27:02.607157 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.606966    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40378a57-440a-4395-98b7-c06dec99dbdb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:27:02.607157 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:02.606977    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40378a57-440a-4395-98b7-c06dec99dbdb-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:27:03.223607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.223572    2562 generic.go:356] "Generic (PLEG): container finished" podID="40378a57-440a-4395-98b7-c06dec99dbdb" containerID="4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70" exitCode=0
Sep 16 19:27:03.223777 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.223629    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"
Sep 16 19:27:03.223777 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.223647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerDied","Data":"4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70"}
Sep 16 19:27:03.223777 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.223704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh" event={"ID":"40378a57-440a-4395-98b7-c06dec99dbdb","Type":"ContainerDied","Data":"1f29d04146c3775466c215f984649cee7e9aeb4747514cbf9cf6edd480a1d6c1"}
Sep 16 19:27:03.223777 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.223731    2562 scope.go:117] "RemoveContainer" containerID="226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf"
Sep 16 19:27:03.232387 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.232368    2562 scope.go:117] "RemoveContainer" containerID="4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70"
Sep 16 19:27:03.239877 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.239856    2562 scope.go:117] "RemoveContainer" containerID="20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c"
Sep 16 19:27:03.247076 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.247053    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"]
Sep 16 19:27:03.248469 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.248448    2562 scope.go:117] "RemoveContainer" containerID="226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf"
Sep 16 19:27:03.249065 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:27:03.249040    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf\": container with ID starting with 226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf not found: ID does not exist" containerID="226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf"
Sep 16 19:27:03.249150 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.249077    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf"} err="failed to get container status \"226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf\": rpc error: code = NotFound desc = could not find container \"226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf\": container with ID starting with 226affef24c7a62471033fd5e01f6e8ce5c821aab90e3368d0c0a65086653ddf not found: ID does not exist"
Sep 16 19:27:03.249150 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.249101    2562 scope.go:117] "RemoveContainer" containerID="4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70"
Sep 16 19:27:03.249426 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:27:03.249401    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70\": container with ID starting with 4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70 not found: ID does not exist" containerID="4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70"
Sep 16 19:27:03.249491 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.249433    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70"} err="failed to get container status \"4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70\": rpc error: code = NotFound desc = could not find container \"4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70\": container with ID starting with 4509527db1143f593c69aa7e2ea6bad74d22fcc733114a37bdacc8ead3f65d70 not found: ID does not exist"
Sep 16 19:27:03.249491 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.249450    2562 scope.go:117] "RemoveContainer" containerID="20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c"
Sep 16 19:27:03.249737 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:27:03.249721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c\": container with ID starting with 20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c not found: ID does not exist" containerID="20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c"
Sep 16 19:27:03.249776 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.249744    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c"} err="failed to get container status \"20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c\": rpc error: code = NotFound desc = could not find container \"20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c\": container with ID starting with 20539821a6dbc75deb60de298c14e8840c1240ea3ae505726fa01f04c2f99b8c not found: ID does not exist"
Sep 16 19:27:03.250461 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:03.250442    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-6k6nh"]
Sep 16 19:27:04.121619 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:04.121588    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" path="/var/lib/kubelet/pods/40378a57-440a-4395-98b7-c06dec99dbdb/volumes"
Sep 16 19:27:06.489409 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:06.489371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:06.635660 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:06.635619    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:10.112725 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:10.112685    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:15.295373 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:15.295337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:15.458905 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:15.458859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:26.664642 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:26.664602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:31.734228 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.734184    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:31.763468 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763438    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"]
Sep 16 19:27:31.763762 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763749    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="storage-initializer"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763763    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="storage-initializer"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763774    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kube-rbac-proxy"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763779    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kube-rbac-proxy"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763785    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="storage-initializer"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763791    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="storage-initializer"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763800    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kube-rbac-proxy"
Sep 16 19:27:31.763805 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763805    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kube-rbac-proxy"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763816    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kserve-container"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763821    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kserve-container"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763835    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763841    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763888    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kserve-container"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763895    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kserve-container"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763901    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="40378a57-440a-4395-98b7-c06dec99dbdb" containerName="kube-rbac-proxy"
Sep 16 19:27:31.764061 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.763909    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ad20e2-bc46-4d3c-8052-35cec08892e5" containerName="kube-rbac-proxy"
Sep 16 19:27:31.766953 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.766935    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.769285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.769249    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 16 19:27:31.769399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.769307    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:27:31.769692 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.769661    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:27:31.770124 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.770108    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:27:31.770235 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.770220    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:27:31.770309 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.770240    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:27:31.770309 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.770221    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:27:31.778037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.778016    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"]
Sep 16 19:27:31.849227 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.849198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.849227 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.849231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e285d915-0dd1-4867-a1ef-c96dff344e53-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.849438 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.849252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.849438 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.849279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875xs\" (UniqueName: \"kubernetes.io/projected/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-api-access-875xs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.849438 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.849420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e285d915-0dd1-4867-a1ef-c96dff344e53-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.950625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.950589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e285d915-0dd1-4867-a1ef-c96dff344e53-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.950625 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.950626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.950821 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.950646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e285d915-0dd1-4867-a1ef-c96dff344e53-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.950926 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.950905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.950976 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.950943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875xs\" (UniqueName: \"kubernetes.io/projected/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-api-access-875xs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.951023 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.950965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e285d915-0dd1-4867-a1ef-c96dff344e53-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.951294 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.951267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.951464 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.951447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.953392 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.953373    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e285d915-0dd1-4867-a1ef-c96dff344e53-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.959722 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.959701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-875xs\" (UniqueName: \"kubernetes.io/projected/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-api-access-875xs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:31.961094 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:31.961073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:32.077073 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:32.077052    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:32.351354 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:32.351310    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"]
Sep 16 19:27:32.353492 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:27:32.353467    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode285d915_0dd1_4867_a1ef_c96dff344e53.slice/crio-e1805a6dc1fc31dc2faf80463634936a439bceefe3794c5f6fb762e273579d65 WatchSource:0}: Error finding container e1805a6dc1fc31dc2faf80463634936a439bceefe3794c5f6fb762e273579d65: Status 404 returned error can't find the container with id e1805a6dc1fc31dc2faf80463634936a439bceefe3794c5f6fb762e273579d65
Sep 16 19:27:33.348903 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:33.348870    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerStarted","Data":"89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda"}
Sep 16 19:27:33.349339 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:33.348932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerStarted","Data":"e1805a6dc1fc31dc2faf80463634936a439bceefe3794c5f6fb762e273579d65"}
Sep 16 19:27:37.365850 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:37.365820    2562 generic.go:356] "Generic (PLEG): container finished" podID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerID="89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda" exitCode=0
Sep 16 19:27:37.366376 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:37.365865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerDied","Data":"89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda"}
Sep 16 19:27:38.372443 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:38.372406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerStarted","Data":"74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341"}
Sep 16 19:27:38.372804 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:38.372475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerStarted","Data":"2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732"}
Sep 16 19:27:38.372804 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:38.372683    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:38.392156 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:38.392100    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" podStartSLOduration=3.301344342 podStartE2EDuration="7.392081171s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.090736829s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:38.390270446 +0000 UTC m=+2660.806973206" watchObservedRunningTime="2026-09-16 19:27:38.392081171 +0000 UTC m=+2660.808783935"
Sep 16 19:27:39.376926 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:39.376900    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:45.386732 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:45.386701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:27:45.826768 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:27:45.826736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:15.390373 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:15.390274    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:28:21.844302 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:21.844269    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"]
Sep 16 19:28:21.844694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:21.844531    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kserve-container" containerID="cri-o://2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732" gracePeriod=30
Sep 16 19:28:21.844694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:21.844613    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kube-rbac-proxy" containerID="cri-o://74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341" gracePeriod=30
Sep 16 19:28:21.851567 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:21.851540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:22.555571 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:22.555540    2562 generic.go:356] "Generic (PLEG): container finished" podID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerID="74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341" exitCode=2
Sep 16 19:28:22.555761 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:22.555617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerDied","Data":"74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341"}
Sep 16 19:28:23.013693 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.013664    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:28:23.188367 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-cabundle-cert\") pod \"e285d915-0dd1-4867-a1ef-c96dff344e53\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") "
Sep 16 19:28:23.188367 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188342    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e285d915-0dd1-4867-a1ef-c96dff344e53-kserve-provision-location\") pod \"e285d915-0dd1-4867-a1ef-c96dff344e53\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") "
Sep 16 19:28:23.188600 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-rbac-proxy-sar-config\") pod \"e285d915-0dd1-4867-a1ef-c96dff344e53\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") "
Sep 16 19:28:23.188600 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188448    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e285d915-0dd1-4867-a1ef-c96dff344e53-proxy-tls\") pod \"e285d915-0dd1-4867-a1ef-c96dff344e53\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") "
Sep 16 19:28:23.188600 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-875xs\" (UniqueName: \"kubernetes.io/projected/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-api-access-875xs\") pod \"e285d915-0dd1-4867-a1ef-c96dff344e53\" (UID: \"e285d915-0dd1-4867-a1ef-c96dff344e53\") "
Sep 16 19:28:23.188772 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188704    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e285d915-0dd1-4867-a1ef-c96dff344e53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e285d915-0dd1-4867-a1ef-c96dff344e53" (UID: "e285d915-0dd1-4867-a1ef-c96dff344e53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:23.188833 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188769    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e285d915-0dd1-4867-a1ef-c96dff344e53" (UID: "e285d915-0dd1-4867-a1ef-c96dff344e53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:23.188881 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.188832    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e285d915-0dd1-4867-a1ef-c96dff344e53" (UID: "e285d915-0dd1-4867-a1ef-c96dff344e53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:23.190786 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.190754    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e285d915-0dd1-4867-a1ef-c96dff344e53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e285d915-0dd1-4867-a1ef-c96dff344e53" (UID: "e285d915-0dd1-4867-a1ef-c96dff344e53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:23.190901 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.190853    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-api-access-875xs" (OuterVolumeSpecName: "kube-api-access-875xs") pod "e285d915-0dd1-4867-a1ef-c96dff344e53" (UID: "e285d915-0dd1-4867-a1ef-c96dff344e53"). InnerVolumeSpecName "kube-api-access-875xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:23.290048 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.290005    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e285d915-0dd1-4867-a1ef-c96dff344e53-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:28:23.290048 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.290048    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875xs\" (UniqueName: \"kubernetes.io/projected/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-api-access-875xs\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:28:23.290248 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.290064    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:28:23.290248 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.290078    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e285d915-0dd1-4867-a1ef-c96dff344e53-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:28:23.290248 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.290091    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e285d915-0dd1-4867-a1ef-c96dff344e53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:28:23.561779 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.561744    2562 generic.go:356] "Generic (PLEG): container finished" podID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerID="2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732" exitCode=0
Sep 16 19:28:23.561952 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.561796    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"
Sep 16 19:28:23.561952 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.561790    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerDied","Data":"2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732"}
Sep 16 19:28:23.561952 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.561925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s" event={"ID":"e285d915-0dd1-4867-a1ef-c96dff344e53","Type":"ContainerDied","Data":"e1805a6dc1fc31dc2faf80463634936a439bceefe3794c5f6fb762e273579d65"}
Sep 16 19:28:23.561952 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.561956    2562 scope.go:117] "RemoveContainer" containerID="74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341"
Sep 16 19:28:23.571024 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.571006    2562 scope.go:117] "RemoveContainer" containerID="2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732"
Sep 16 19:28:23.578841 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.578822    2562 scope.go:117] "RemoveContainer" containerID="89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda"
Sep 16 19:28:23.583683 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.583663    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"]
Sep 16 19:28:23.586452 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.586438    2562 scope.go:117] "RemoveContainer" containerID="74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341"
Sep 16 19:28:23.586725 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:28:23.586708    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341\": container with ID starting with 74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341 not found: ID does not exist" containerID="74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341"
Sep 16 19:28:23.586774 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.586734    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341"} err="failed to get container status \"74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341\": rpc error: code = NotFound desc = could not find container \"74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341\": container with ID starting with 74d7c80f4252a6dba5b1d9e2d38fb3789c58835f8658348bd98be09e7ea02341 not found: ID does not exist"
Sep 16 19:28:23.586774 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.586750    2562 scope.go:117] "RemoveContainer" containerID="2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732"
Sep 16 19:28:23.586991 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:28:23.586972    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732\": container with ID starting with 2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732 not found: ID does not exist" containerID="2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732"
Sep 16 19:28:23.587058 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.587004    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732"} err="failed to get container status \"2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732\": rpc error: code = NotFound desc = could not find container \"2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732\": container with ID starting with 2fd0668e6dd33e68fb1bbcc9185fed10a75b149f063acc781e04ce8b9f743732 not found: ID does not exist"
Sep 16 19:28:23.587058 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.587025    2562 scope.go:117] "RemoveContainer" containerID="89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda"
Sep 16 19:28:23.587271 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:28:23.587249    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda\": container with ID starting with 89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda not found: ID does not exist" containerID="89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda"
Sep 16 19:28:23.587363 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.587275    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda"} err="failed to get container status \"89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda\": rpc error: code = NotFound desc = could not find container \"89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda\": container with ID starting with 89a5a8191a516e1f3d317a901df4078ce0ad7502d66aef6aa87958a156688bda not found: ID does not exist"
Sep 16 19:28:23.589655 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:23.589634    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-l4d7s"]
Sep 16 19:28:24.122263 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:24.122231    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" path="/var/lib/kubelet/pods/e285d915-0dd1-4867-a1ef-c96dff344e53/volumes"
Sep 16 19:28:26.976732 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:26.976698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:27.023675 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.023646    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"]
Sep 16 19:28:27.023965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.023952    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kube-rbac-proxy"
Sep 16 19:28:27.023965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.023966    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kube-rbac-proxy"
Sep 16 19:28:27.024056 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.023977    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="storage-initializer"
Sep 16 19:28:27.024056 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.023983    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="storage-initializer"
Sep 16 19:28:27.024056 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.024001    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kserve-container"
Sep 16 19:28:27.024056 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.024007    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kserve-container"
Sep 16 19:28:27.024180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.024061    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kserve-container"
Sep 16 19:28:27.024180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.024071    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e285d915-0dd1-4867-a1ef-c96dff344e53" containerName="kube-rbac-proxy"
Sep 16 19:28:27.028948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.028928    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.031123 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.031099    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:27.031289 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.031270    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:28:27.031444 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.031404    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:28:27.031578 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.031562    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:28:27.031623 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.031598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 16 19:28:27.032001 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.031984    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:28:27.032071 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.032009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 16 19:28:27.044985 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.044961    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"]
Sep 16 19:28:27.120232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.120200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.120232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.120236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6m5\" (UniqueName: \"kubernetes.io/projected/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-api-access-9f6m5\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.120506 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.120295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1059ff1-dd54-4114-9df2-c1a335730c93-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.120506 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.120386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.120506 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.120455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221507 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221507 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6m5\" (UniqueName: \"kubernetes.io/projected/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-api-access-9f6m5\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221507 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1059ff1-dd54-4114-9df2-c1a335730c93-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221507 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221507 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:28:27.221430    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 19:28:27.221794 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:28:27.221529    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls podName:d1059ff1-dd54-4114-9df2-c1a335730c93 nodeName:}" failed. No retries permitted until 2026-09-16 19:28:27.721508015 +0000 UTC m=+2710.138210753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls") pod "isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" (UID: "d1059ff1-dd54-4114-9df2-c1a335730c93") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 19:28:27.221794 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1059ff1-dd54-4114-9df2-c1a335730c93-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.221995 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.222030 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.221987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.232887 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.232825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6m5\" (UniqueName: \"kubernetes.io/projected/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-api-access-9f6m5\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.327799 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.327763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:27.726147 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.726104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.728760 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.728728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:27.939336 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:27.939284    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:28.215914 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:28.215889    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"]
Sep 16 19:28:28.217985 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:28:28.217958    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1059ff1_dd54_4114_9df2_c1a335730c93.slice/crio-e2f588ba371a1934a0c8e8f96d19def4061a5241901ba32b253fa988debf7498 WatchSource:0}: Error finding container e2f588ba371a1934a0c8e8f96d19def4061a5241901ba32b253fa988debf7498: Status 404 returned error can't find the container with id e2f588ba371a1934a0c8e8f96d19def4061a5241901ba32b253fa988debf7498
Sep 16 19:28:28.583864 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:28.583835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerStarted","Data":"398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52"}
Sep 16 19:28:28.584037 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:28.583899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerStarted","Data":"e2f588ba371a1934a0c8e8f96d19def4061a5241901ba32b253fa988debf7498"}
Sep 16 19:28:32.602586 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:32.602555    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerID="398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52" exitCode=0
Sep 16 19:28:32.602981 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:32.602599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerDied","Data":"398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52"}
Sep 16 19:28:33.611947 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:33.611908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerStarted","Data":"ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438"}
Sep 16 19:28:35.624916 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:35.624887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerStarted","Data":"d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2"}
Sep 16 19:28:35.625271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:35.624955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerStarted","Data":"1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c"}
Sep 16 19:28:35.625271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:35.625054    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:35.625271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:35.625107    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:35.625271 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:35.625131    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:28:35.646676 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:35.646635    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podStartSLOduration=2.043195324 podStartE2EDuration="8.646621321s" totalImagesPullingTime="2.826973667s" totalInitContainerRuntime="3.77645233s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:35.6465495 +0000 UTC m=+2718.063252259" watchObservedRunningTime="2026-09-16 19:28:35.646621321 +0000 UTC m=+2718.063324080"
Sep 16 19:28:39.448764 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:39.448727    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:39.579656 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:39.579561    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:39.910850 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:39.910765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:41.635003 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:28:41.634978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:29:07.940432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:07.940394    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:29:11.638961 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:11.638928    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:29:41.640170 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:41.640141    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:29:47.083743 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.083709    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"]
Sep 16 19:29:47.084185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.083976    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container" containerID="cri-o://ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438" gracePeriod=30
Sep 16 19:29:47.084185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.084019    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" containerID="cri-o://d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2" gracePeriod=30
Sep 16 19:29:47.084185 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.084029    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-agent" containerID="cri-o://1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c" gracePeriod=30
Sep 16 19:29:47.105344 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.105292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:47.929458 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.929421    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerID="d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2" exitCode=2
Sep 16 19:29:47.929660 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:47.929471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerDied","Data":"d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2"}
Sep 16 19:29:49.940259 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:49.940226    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerID="ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438" exitCode=0
Sep 16 19:29:49.940652 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:49.940278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerDied","Data":"ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438"}
Sep 16 19:29:51.630439 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:51.630394    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:29:51.635470 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:51.635444    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:29:52.216141 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:52.216108    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:52.381264 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:52.381226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:56.630772 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:29:56.630734    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:30:01.630654 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:01.630603    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:30:01.631072 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:01.630783    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:30:01.635486 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:01.635461    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:30:06.630168 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:06.630131    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:30:09.682413 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:09.682374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:10.745002 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.744960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:10.760919 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.760895    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"]
Sep 16 19:30:10.764383 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.764367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:10.766777 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.766743    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:30:10.766921 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.766740    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:30:10.766921 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.766820    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 16 19:30:10.775396 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.775371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"]
Sep 16 19:30:10.908404 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.908366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:10.908579 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.908421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76dkr\" (UniqueName: \"kubernetes.io/projected/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-api-access-76dkr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:10.908579 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.908486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2db64e-196d-4e16-89d4-4cdf89085778-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:10.908579 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:10.908570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.009186 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.009103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76dkr\" (UniqueName: \"kubernetes.io/projected/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-api-access-76dkr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.009186 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.009146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2db64e-196d-4e16-89d4-4cdf89085778-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.009432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.009201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.009432 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.009256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.009432 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:11.009374    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 16 19:30:11.009570 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:11.009438    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls podName:0e2db64e-196d-4e16-89d4-4cdf89085778 nodeName:}" failed. No retries permitted until 2026-09-16 19:30:11.509423465 +0000 UTC m=+2813.926126202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" (UID: "0e2db64e-196d-4e16-89d4-4cdf89085778") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 16 19:30:11.009740 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.009718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2db64e-196d-4e16-89d4-4cdf89085778-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.009958 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.009943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.021083 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.021026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76dkr\" (UniqueName: \"kubernetes.io/projected/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-api-access-76dkr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.160044 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.160005    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:11.513412 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.513371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.515966 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.515939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.630220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.630174    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:30:11.635622 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.635583    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.58:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 16 19:30:11.635742 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.635703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:30:11.674577 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.674541    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:11.947272 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:11.947238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"]
Sep 16 19:30:11.956065 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:30:11.956036    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2db64e_196d_4e16_89d4_4cdf89085778.slice/crio-82d6d59ae7c93a07f4574e91871793a881e174e0ea82fa68d305e730ffd5b7b7 WatchSource:0}: Error finding container 82d6d59ae7c93a07f4574e91871793a881e174e0ea82fa68d305e730ffd5b7b7: Status 404 returned error can't find the container with id 82d6d59ae7c93a07f4574e91871793a881e174e0ea82fa68d305e730ffd5b7b7
Sep 16 19:30:12.031113 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:12.031087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" event={"ID":"0e2db64e-196d-4e16-89d4-4cdf89085778","Type":"ContainerStarted","Data":"a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9"}
Sep 16 19:30:12.031232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:12.031140    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" event={"ID":"0e2db64e-196d-4e16-89d4-4cdf89085778","Type":"ContainerStarted","Data":"82d6d59ae7c93a07f4574e91871793a881e174e0ea82fa68d305e730ffd5b7b7"}
Sep 16 19:30:16.630264 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:16.630219    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 16 19:30:17.053220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.053191    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7_0e2db64e-196d-4e16-89d4-4cdf89085778/storage-initializer/0.log"
Sep 16 19:30:17.053403 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.053232    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerID="a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9" exitCode=1
Sep 16 19:30:17.053403 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.053271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" event={"ID":"0e2db64e-196d-4e16-89d4-4cdf89085778","Type":"ContainerDied","Data":"a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9"}
Sep 16 19:30:17.236977 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.236953    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:30:17.262232 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262198    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls\") pod \"d1059ff1-dd54-4114-9df2-c1a335730c93\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") "
Sep 16 19:30:17.262523 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262259    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f6m5\" (UniqueName: \"kubernetes.io/projected/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-api-access-9f6m5\") pod \"d1059ff1-dd54-4114-9df2-c1a335730c93\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") "
Sep 16 19:30:17.262523 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262297    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-cabundle-cert\") pod \"d1059ff1-dd54-4114-9df2-c1a335730c93\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") "
Sep 16 19:30:17.262523 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262350    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-rbac-proxy-sar-config\") pod \"d1059ff1-dd54-4114-9df2-c1a335730c93\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") "
Sep 16 19:30:17.262523 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1059ff1-dd54-4114-9df2-c1a335730c93-kserve-provision-location\") pod \"d1059ff1-dd54-4114-9df2-c1a335730c93\" (UID: \"d1059ff1-dd54-4114-9df2-c1a335730c93\") "
Sep 16 19:30:17.263376 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262905    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1059ff1-dd54-4114-9df2-c1a335730c93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1059ff1-dd54-4114-9df2-c1a335730c93" (UID: "d1059ff1-dd54-4114-9df2-c1a335730c93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:17.263376 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.262949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1059ff1-dd54-4114-9df2-c1a335730c93" (UID: "d1059ff1-dd54-4114-9df2-c1a335730c93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:17.263752 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.263681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1059ff1-dd54-4114-9df2-c1a335730c93" (UID: "d1059ff1-dd54-4114-9df2-c1a335730c93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:17.265407 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.265383    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1059ff1-dd54-4114-9df2-c1a335730c93" (UID: "d1059ff1-dd54-4114-9df2-c1a335730c93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:17.265680 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.265615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-api-access-9f6m5" (OuterVolumeSpecName: "kube-api-access-9f6m5") pod "d1059ff1-dd54-4114-9df2-c1a335730c93" (UID: "d1059ff1-dd54-4114-9df2-c1a335730c93"). InnerVolumeSpecName "kube-api-access-9f6m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:17.363837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.363755    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1059ff1-dd54-4114-9df2-c1a335730c93-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:17.363837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.363793    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1059ff1-dd54-4114-9df2-c1a335730c93-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:17.363837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.363809    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6m5\" (UniqueName: \"kubernetes.io/projected/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-api-access-9f6m5\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:17.363837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.363822    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-cabundle-cert\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:17.363837 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:17.363835    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1059ff1-dd54-4114-9df2-c1a335730c93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:18.059816 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.059783    2562 generic.go:356] "Generic (PLEG): container finished" podID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerID="1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c" exitCode=137
Sep 16 19:30:18.060260 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.059840    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"
Sep 16 19:30:18.060260 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.059863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerDied","Data":"1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c"}
Sep 16 19:30:18.060260 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.059945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc" event={"ID":"d1059ff1-dd54-4114-9df2-c1a335730c93","Type":"ContainerDied","Data":"e2f588ba371a1934a0c8e8f96d19def4061a5241901ba32b253fa988debf7498"}
Sep 16 19:30:18.060260 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.059966    2562 scope.go:117] "RemoveContainer" containerID="d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2"
Sep 16 19:30:18.061957 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.061937    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7_0e2db64e-196d-4e16-89d4-4cdf89085778/storage-initializer/0.log"
Sep 16 19:30:18.062075 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.062012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" event={"ID":"0e2db64e-196d-4e16-89d4-4cdf89085778","Type":"ContainerStarted","Data":"37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891"}
Sep 16 19:30:18.069343 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.069306    2562 scope.go:117] "RemoveContainer" containerID="1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c"
Sep 16 19:30:18.076851 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.076832    2562 scope.go:117] "RemoveContainer" containerID="ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438"
Sep 16 19:30:18.084094 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.084076    2562 scope.go:117] "RemoveContainer" containerID="398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52"
Sep 16 19:30:18.092208 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.092192    2562 scope.go:117] "RemoveContainer" containerID="d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2"
Sep 16 19:30:18.092479 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:18.092459    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2\": container with ID starting with d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2 not found: ID does not exist" containerID="d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2"
Sep 16 19:30:18.092527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.092487    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2"} err="failed to get container status \"d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2\": rpc error: code = NotFound desc = could not find container \"d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2\": container with ID starting with d352c2e6524d2e97919f2368d6f71faf124fc2196a736722274664fc0d3e4cd2 not found: ID does not exist"
Sep 16 19:30:18.092527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.092503    2562 scope.go:117] "RemoveContainer" containerID="1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c"
Sep 16 19:30:18.092736 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:18.092721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c\": container with ID starting with 1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c not found: ID does not exist" containerID="1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c"
Sep 16 19:30:18.092784 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.092743    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c"} err="failed to get container status \"1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c\": rpc error: code = NotFound desc = could not find container \"1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c\": container with ID starting with 1b117d8be41fa88af31f6c34aa3b43d0d883ae9df7da62dc604caf59b14b067c not found: ID does not exist"
Sep 16 19:30:18.092784 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.092761    2562 scope.go:117] "RemoveContainer" containerID="ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438"
Sep 16 19:30:18.092982 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:18.092963    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438\": container with ID starting with ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438 not found: ID does not exist" containerID="ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438"
Sep 16 19:30:18.093041 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.092992    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438"} err="failed to get container status \"ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438\": rpc error: code = NotFound desc = could not find container \"ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438\": container with ID starting with ccc4d1b7cf21c48fa45289339440ad817eedd91f437d16ec5009e5bcf207f438 not found: ID does not exist"
Sep 16 19:30:18.093041 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.093013    2562 scope.go:117] "RemoveContainer" containerID="398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52"
Sep 16 19:30:18.093273 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:18.093259    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52\": container with ID starting with 398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52 not found: ID does not exist" containerID="398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52"
Sep 16 19:30:18.093312 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.093278    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52"} err="failed to get container status \"398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52\": rpc error: code = NotFound desc = could not find container \"398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52\": container with ID starting with 398e2f51ffc7e5845e9c296edb56c50d0d27d47cf91123ad3ded6e20e9ed8f52 not found: ID does not exist"
Sep 16 19:30:18.101236 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.101215    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"]
Sep 16 19:30:18.105278 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.105256    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-677dfdb8d8-4drgc"]
Sep 16 19:30:18.123189 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:18.123160    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" path="/var/lib/kubelet/pods/d1059ff1-dd54-4114-9df2-c1a335730c93/volumes"
Sep 16 19:30:20.750128 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:20.750097    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"]
Sep 16 19:30:20.750556 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:20.750351    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer" containerID="cri-o://37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891" gracePeriod=30
Sep 16 19:30:20.769845 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:20.769809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:21.693179 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.693157    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7_0e2db64e-196d-4e16-89d4-4cdf89085778/storage-initializer/1.log"
Sep 16 19:30:21.693587 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.693569    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7_0e2db64e-196d-4e16-89d4-4cdf89085778/storage-initializer/0.log"
Sep 16 19:30:21.696073 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.696054    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:21.800195 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.800165    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76dkr\" (UniqueName: \"kubernetes.io/projected/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-api-access-76dkr\") pod \"0e2db64e-196d-4e16-89d4-4cdf89085778\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") "
Sep 16 19:30:21.800589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.800230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2db64e-196d-4e16-89d4-4cdf89085778-kserve-provision-location\") pod \"0e2db64e-196d-4e16-89d4-4cdf89085778\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") "
Sep 16 19:30:21.800589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.800276    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-rbac-proxy-sar-config\") pod \"0e2db64e-196d-4e16-89d4-4cdf89085778\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") "
Sep 16 19:30:21.800589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.800333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls\") pod \"0e2db64e-196d-4e16-89d4-4cdf89085778\" (UID: \"0e2db64e-196d-4e16-89d4-4cdf89085778\") "
Sep 16 19:30:21.800589 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.800558    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2db64e-196d-4e16-89d4-4cdf89085778-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e2db64e-196d-4e16-89d4-4cdf89085778" (UID: "0e2db64e-196d-4e16-89d4-4cdf89085778"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:21.800733 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.800615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e2db64e-196d-4e16-89d4-4cdf89085778" (UID: "0e2db64e-196d-4e16-89d4-4cdf89085778"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:21.802537 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.802513    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-api-access-76dkr" (OuterVolumeSpecName: "kube-api-access-76dkr") pod "0e2db64e-196d-4e16-89d4-4cdf89085778" (UID: "0e2db64e-196d-4e16-89d4-4cdf89085778"). InnerVolumeSpecName "kube-api-access-76dkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:21.802603 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.802548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e2db64e-196d-4e16-89d4-4cdf89085778" (UID: "0e2db64e-196d-4e16-89d4-4cdf89085778"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:21.850882 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.850847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:21.901077 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.901046    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76dkr\" (UniqueName: \"kubernetes.io/projected/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-api-access-76dkr\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:21.901077 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.901070    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e2db64e-196d-4e16-89d4-4cdf89085778-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:21.901077 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.901079    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e2db64e-196d-4e16-89d4-4cdf89085778-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:21.901307 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:21.901089    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e2db64e-196d-4e16-89d4-4cdf89085778-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:30:22.088694 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.088612    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7_0e2db64e-196d-4e16-89d4-4cdf89085778/storage-initializer/1.log"
Sep 16 19:30:22.089009 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.088985    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7_0e2db64e-196d-4e16-89d4-4cdf89085778/storage-initializer/0.log"
Sep 16 19:30:22.089124 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.089026    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerID="37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891" exitCode=1
Sep 16 19:30:22.089124 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.089080    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"
Sep 16 19:30:22.089204 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.089118    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" event={"ID":"0e2db64e-196d-4e16-89d4-4cdf89085778","Type":"ContainerDied","Data":"37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891"}
Sep 16 19:30:22.089204 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.089184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7" event={"ID":"0e2db64e-196d-4e16-89d4-4cdf89085778","Type":"ContainerDied","Data":"82d6d59ae7c93a07f4574e91871793a881e174e0ea82fa68d305e730ffd5b7b7"}
Sep 16 19:30:22.089275 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.089224    2562 scope.go:117] "RemoveContainer" containerID="37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891"
Sep 16 19:30:22.098216 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.098198    2562 scope.go:117] "RemoveContainer" containerID="a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9"
Sep 16 19:30:22.105980 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.105965    2562 scope.go:117] "RemoveContainer" containerID="37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891"
Sep 16 19:30:22.106233 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:22.106216    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891\": container with ID starting with 37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891 not found: ID does not exist" containerID="37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891"
Sep 16 19:30:22.106285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.106241    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891"} err="failed to get container status \"37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891\": rpc error: code = NotFound desc = could not find container \"37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891\": container with ID starting with 37bfa66fa1fc8be6fe2e86f61780e8e0112c66aab617119dab00b860c2db6891 not found: ID does not exist"
Sep 16 19:30:22.106285 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.106255    2562 scope.go:117] "RemoveContainer" containerID="a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9"
Sep 16 19:30:22.106586 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:30:22.106565    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9\": container with ID starting with a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9 not found: ID does not exist" containerID="a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9"
Sep 16 19:30:22.106630 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.106594    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9"} err="failed to get container status \"a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9\": rpc error: code = NotFound desc = could not find container \"a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9\": container with ID starting with a7ed1c3da97a3727aa82c622876d68cc786e191065c689ebd7303a3d02ad8de9 not found: ID does not exist"
Sep 16 19:30:22.108354 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.108307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:22.125873 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.125850    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"]
Sep 16 19:30:22.132410 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:22.132385    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-79b5dd6549-hd7b7"]
Sep 16 19:30:24.121813 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:30:24.121775    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" path="/var/lib/kubelet/pods/0e2db64e-196d-4e16-89d4-4cdf89085778/volumes"
Sep 16 19:31:03.478636 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:03.478592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:08.511914 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:08.511879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:10.047438 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:10.047348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:41.995860 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:41.995823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:42.975534 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:42.975499    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:43.009627 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009595    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"]
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009908    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009920    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009929    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009934    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009941    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009947    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009952    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009957    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer"
Sep 16 19:31:43.009965 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009968    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="storage-initializer"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009973    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="storage-initializer"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009980    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-agent"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.009985    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-agent"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.010038    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-agent"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.010047    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.010063    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kserve-container"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.010069    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1059ff1-dd54-4114-9df2-c1a335730c93" containerName="kube-rbac-proxy"
Sep 16 19:31:43.010293 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.010166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2db64e-196d-4e16-89d4-4cdf89085778" containerName="storage-initializer"
Sep 16 19:31:43.013125 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.013105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.015883 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.015857    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 16 19:31:43.015992 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.015918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:31:43.015992 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.015958    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:31:43.016079 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.015962    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:31:43.016207 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.016192    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5nd49\""
Sep 16 19:31:43.016683 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.016664    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:31:43.030006 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.029982    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"]
Sep 16 19:31:43.055147 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.055119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.055272 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.055151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bllf\" (UniqueName: \"kubernetes.io/projected/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-api-access-5bllf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.055272 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.055192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801949bf-42ed-4a53-bcd9-eaef94218de8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.055272 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.055241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801949bf-42ed-4a53-bcd9-eaef94218de8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.155920 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.155883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.155920 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.155918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bllf\" (UniqueName: \"kubernetes.io/projected/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-api-access-5bllf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.156155 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.155950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801949bf-42ed-4a53-bcd9-eaef94218de8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.156155 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.155973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801949bf-42ed-4a53-bcd9-eaef94218de8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.156419 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.156400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801949bf-42ed-4a53-bcd9-eaef94218de8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.156645 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.156625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.158587 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.158568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801949bf-42ed-4a53-bcd9-eaef94218de8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.165249 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.165223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:43.165938 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.165921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bllf\" (UniqueName: \"kubernetes.io/projected/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-api-access-5bllf\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.323160 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.323130    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:43.596948 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:43.596925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"]
Sep 16 19:31:43.599625 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:31:43.599599    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801949bf_42ed_4a53_bcd9_eaef94218de8.slice/crio-c675d84056837431fece1e8a30073ff1ae92f13217d3f1bf367726f9fcad6bd3 WatchSource:0}: Error finding container c675d84056837431fece1e8a30073ff1ae92f13217d3f1bf367726f9fcad6bd3: Status 404 returned error can't find the container with id c675d84056837431fece1e8a30073ff1ae92f13217d3f1bf367726f9fcad6bd3
Sep 16 19:31:44.436747 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:44.436710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" event={"ID":"801949bf-42ed-4a53-bcd9-eaef94218de8","Type":"ContainerStarted","Data":"82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013"}
Sep 16 19:31:44.437214 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:44.436774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" event={"ID":"801949bf-42ed-4a53-bcd9-eaef94218de8","Type":"ContainerStarted","Data":"c675d84056837431fece1e8a30073ff1ae92f13217d3f1bf367726f9fcad6bd3"}
Sep 16 19:31:50.465136 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:50.465061    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5_801949bf-42ed-4a53-bcd9-eaef94218de8/storage-initializer/0.log"
Sep 16 19:31:50.465136 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:50.465100    2562 generic.go:356] "Generic (PLEG): container finished" podID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerID="82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013" exitCode=1
Sep 16 19:31:50.465566 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:50.465180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" event={"ID":"801949bf-42ed-4a53-bcd9-eaef94218de8","Type":"ContainerDied","Data":"82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013"}
Sep 16 19:31:51.471853 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:51.471827    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5_801949bf-42ed-4a53-bcd9-eaef94218de8/storage-initializer/0.log"
Sep 16 19:31:51.472220 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:51.471892    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" event={"ID":"801949bf-42ed-4a53-bcd9-eaef94218de8","Type":"ContainerStarted","Data":"4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3"}
Sep 16 19:31:53.003273 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:53.003228    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"]
Sep 16 19:31:53.003803 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:53.003463    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer" containerID="cri-o://4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3" gracePeriod=30
Sep 16 19:31:53.029093 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:53.029062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:54.074763 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:54.074730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:54.177036 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:54.176998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:55.341703 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.341682    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5_801949bf-42ed-4a53-bcd9-eaef94218de8/storage-initializer/1.log"
Sep 16 19:31:55.342109 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.342090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5_801949bf-42ed-4a53-bcd9-eaef94218de8/storage-initializer/0.log"
Sep 16 19:31:55.344482 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.344464    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:55.464368 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.464260    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bllf\" (UniqueName: \"kubernetes.io/projected/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-api-access-5bllf\") pod \"801949bf-42ed-4a53-bcd9-eaef94218de8\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") "
Sep 16 19:31:55.464368 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.464335    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-rbac-proxy-sar-config\") pod \"801949bf-42ed-4a53-bcd9-eaef94218de8\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") "
Sep 16 19:31:55.464368 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.464365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801949bf-42ed-4a53-bcd9-eaef94218de8-proxy-tls\") pod \"801949bf-42ed-4a53-bcd9-eaef94218de8\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") "
Sep 16 19:31:55.464595 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.464408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801949bf-42ed-4a53-bcd9-eaef94218de8-kserve-provision-location\") pod \"801949bf-42ed-4a53-bcd9-eaef94218de8\" (UID: \"801949bf-42ed-4a53-bcd9-eaef94218de8\") "
Sep 16 19:31:55.464723 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.464698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801949bf-42ed-4a53-bcd9-eaef94218de8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "801949bf-42ed-4a53-bcd9-eaef94218de8" (UID: "801949bf-42ed-4a53-bcd9-eaef94218de8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:55.464794 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.464774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "801949bf-42ed-4a53-bcd9-eaef94218de8" (UID: "801949bf-42ed-4a53-bcd9-eaef94218de8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:55.466647 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.466625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801949bf-42ed-4a53-bcd9-eaef94218de8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "801949bf-42ed-4a53-bcd9-eaef94218de8" (UID: "801949bf-42ed-4a53-bcd9-eaef94218de8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:55.466747 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.466643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-api-access-5bllf" (OuterVolumeSpecName: "kube-api-access-5bllf") pod "801949bf-42ed-4a53-bcd9-eaef94218de8" (UID: "801949bf-42ed-4a53-bcd9-eaef94218de8"). InnerVolumeSpecName "kube-api-access-5bllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:55.489160 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489137    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5_801949bf-42ed-4a53-bcd9-eaef94218de8/storage-initializer/1.log"
Sep 16 19:31:55.489582 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489560    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5_801949bf-42ed-4a53-bcd9-eaef94218de8/storage-initializer/0.log"
Sep 16 19:31:55.489652 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489603    2562 generic.go:356] "Generic (PLEG): container finished" podID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerID="4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3" exitCode=1
Sep 16 19:31:55.489693 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" event={"ID":"801949bf-42ed-4a53-bcd9-eaef94218de8","Type":"ContainerDied","Data":"4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3"}
Sep 16 19:31:55.489693 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489658    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"
Sep 16 19:31:55.489765 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489699    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5" event={"ID":"801949bf-42ed-4a53-bcd9-eaef94218de8","Type":"ContainerDied","Data":"c675d84056837431fece1e8a30073ff1ae92f13217d3f1bf367726f9fcad6bd3"}
Sep 16 19:31:55.489765 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.489727    2562 scope.go:117] "RemoveContainer" containerID="4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3"
Sep 16 19:31:55.498421 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.498405    2562 scope.go:117] "RemoveContainer" containerID="82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013"
Sep 16 19:31:55.506397 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.506377    2562 scope.go:117] "RemoveContainer" containerID="4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3"
Sep 16 19:31:55.506667 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:31:55.506648    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3\": container with ID starting with 4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3 not found: ID does not exist" containerID="4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3"
Sep 16 19:31:55.506713 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.506676    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3"} err="failed to get container status \"4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3\": rpc error: code = NotFound desc = could not find container \"4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3\": container with ID starting with 4dc1aad8cae4fe35695a454e9ad892c2cc5204f279538038c5cbe081e1bbc2d3 not found: ID does not exist"
Sep 16 19:31:55.506713 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.506695    2562 scope.go:117] "RemoveContainer" containerID="82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013"
Sep 16 19:31:55.506957 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:31:55.506938    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013\": container with ID starting with 82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013 not found: ID does not exist" containerID="82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013"
Sep 16 19:31:55.507025 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.506962    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013"} err="failed to get container status \"82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013\": rpc error: code = NotFound desc = could not find container \"82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013\": container with ID starting with 82fc698d1d27cc3a515cad8c2027204fb5b80ae6eb9a6a07ceb972ee9ca85013 not found: ID does not exist"
Sep 16 19:31:55.534518 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.534492    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"]
Sep 16 19:31:55.539374 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.539349    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7464556c85-6hqp5"]
Sep 16 19:31:55.565651 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.565623    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bllf\" (UniqueName: \"kubernetes.io/projected/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-api-access-5bllf\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:31:55.565651 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.565646    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/801949bf-42ed-4a53-bcd9-eaef94218de8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:31:55.565791 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.565657    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/801949bf-42ed-4a53-bcd9-eaef94218de8-proxy-tls\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:31:55.565791 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:55.565667    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/801949bf-42ed-4a53-bcd9-eaef94218de8-kserve-provision-location\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:31:56.122118 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:31:56.122085    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" path="/var/lib/kubelet/pods/801949bf-42ed-4a53-bcd9-eaef94218de8/volumes"
Sep 16 19:32:09.670906 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:32:09.670863    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:14.799257 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:32:14.799221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:15.009690 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:32:15.009656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:35.437543 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:32:35.437505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:11.245969 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:11.245927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:16.283721 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:16.283663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:21.382117 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:21.382069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:21.568253 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:21.568218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:24.139724 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:24.139693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:25.215688 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:25.215646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:25.444907 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:25.444871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:35.242525 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:35.242492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:36.380391 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:36.380351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:36.583233 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:33:36.583188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:51.695358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:34:51.695305    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:02.632696 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:02.632665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:07.763973 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:07.763929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:08.015966 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:08.015874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:17.445407 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:17.445372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:22.579598 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:22.579562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:22.748103 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:22.748067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:33.649439 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:35:33.649398    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:42.961003 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:36:42.960965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:44.255384 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:36:44.255350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:53.734963 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:36:53.734906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:59.039184 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:36:59.039146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:59.139076 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:36:59.139037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:05.007967 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.007934    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kzxkd/must-gather-8lhk9"]
Sep 16 19:37:05.008339 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.008262    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer"
Sep 16 19:37:05.008339 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.008276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer"
Sep 16 19:37:05.008420 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.008374    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer"
Sep 16 19:37:05.008420 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.008385    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer"
Sep 16 19:37:05.008483 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.008449    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer"
Sep 16 19:37:05.008483 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.008456    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="801949bf-42ed-4a53-bcd9-eaef94218de8" containerName="storage-initializer"
Sep 16 19:37:05.011519 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.011500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.013883 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.013860    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzxkd\"/\"kube-root-ca.crt\""
Sep 16 19:37:05.014216 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.014197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kzxkd\"/\"default-dockercfg-tcd7n\""
Sep 16 19:37:05.014396 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.014246    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzxkd\"/\"openshift-service-ca.crt\""
Sep 16 19:37:05.014732 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.014711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3420e696-2017-4746-b766-a8655e2c7e27-must-gather-output\") pod \"must-gather-8lhk9\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") " pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.014804 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.014761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzmv\" (UniqueName: \"kubernetes.io/projected/3420e696-2017-4746-b766-a8655e2c7e27-kube-api-access-txzmv\") pod \"must-gather-8lhk9\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") " pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.018784 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.018758    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzxkd/must-gather-8lhk9"]
Sep 16 19:37:05.026780 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.026750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:05.115964 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.115932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3420e696-2017-4746-b766-a8655e2c7e27-must-gather-output\") pod \"must-gather-8lhk9\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") " pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.116137 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.115973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzmv\" (UniqueName: \"kubernetes.io/projected/3420e696-2017-4746-b766-a8655e2c7e27-kube-api-access-txzmv\") pod \"must-gather-8lhk9\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") " pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.116443 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.116425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3420e696-2017-4746-b766-a8655e2c7e27-must-gather-output\") pod \"must-gather-8lhk9\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") " pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.124186 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.124157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzmv\" (UniqueName: \"kubernetes.io/projected/3420e696-2017-4746-b766-a8655e2c7e27-kube-api-access-txzmv\") pod \"must-gather-8lhk9\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") " pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.127062 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.126876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:05.322549 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.322519    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:05.594799 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.594771    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzxkd/must-gather-8lhk9"]
Sep 16 19:37:05.597203 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:37:05.597181    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3420e696_2017_4746_b766_a8655e2c7e27.slice/crio-661f3b902ec6886d1c5be9c4c382a81883d6d91ce565e4b80b2feb548a1bc728 WatchSource:0}: Error finding container 661f3b902ec6886d1c5be9c4c382a81883d6d91ce565e4b80b2feb548a1bc728: Status 404 returned error can't find the container with id 661f3b902ec6886d1c5be9c4c382a81883d6d91ce565e4b80b2feb548a1bc728
Sep 16 19:37:05.599361 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.599342    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:37:05.769719 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:05.769676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzxkd/must-gather-8lhk9" event={"ID":"3420e696-2017-4746-b766-a8655e2c7e27","Type":"ContainerStarted","Data":"661f3b902ec6886d1c5be9c4c382a81883d6d91ce565e4b80b2feb548a1bc728"}
Sep 16 19:37:12.807672 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:12.807627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzxkd/must-gather-8lhk9" event={"ID":"3420e696-2017-4746-b766-a8655e2c7e27","Type":"ContainerStarted","Data":"a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b"}
Sep 16 19:37:12.807672 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:12.807677    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzxkd/must-gather-8lhk9" event={"ID":"3420e696-2017-4746-b766-a8655e2c7e27","Type":"ContainerStarted","Data":"e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"}
Sep 16 19:37:12.823662 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:12.823599    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kzxkd/must-gather-8lhk9" podStartSLOduration=2.656587505 podStartE2EDuration="8.823577979s" totalImagesPullingTime="6.166990474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:12.822627824 +0000 UTC m=+3235.239330608" watchObservedRunningTime="2026-09-16 19:37:12.823577979 +0000 UTC m=+3235.240280741"
Sep 16 19:37:39.225562 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:39.225513    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:39.326566 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:39.326532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:39.941055 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:39.941023    2562 generic.go:356] "Generic (PLEG): container finished" podID="3420e696-2017-4746-b766-a8655e2c7e27" containerID="e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9" exitCode=0
Sep 16 19:37:39.941261 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:39.941105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzxkd/must-gather-8lhk9" event={"ID":"3420e696-2017-4746-b766-a8655e2c7e27","Type":"ContainerDied","Data":"e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"}
Sep 16 19:37:39.941461 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:39.941448    2562 scope.go:117] "RemoveContainer" containerID="e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"
Sep 16 19:37:40.786950 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:40.786924    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kzxkd_must-gather-8lhk9_3420e696-2017-4746-b766-a8655e2c7e27/gather/0.log"
Sep 16 19:37:41.348739 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:41.348702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:41.554339 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:41.554287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:41.647409 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:41.647313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:44.470207 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:44.470171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-89j2l_4b12fc6d-a87c-42cd-be81-b8ff0f0d4fab/global-pull-secret-syncer/0.log"
Sep 16 19:37:44.655240 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:44.655208    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log"
Sep 16 19:37:44.766904 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:44.766822    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-180.ec2.internal_73b2ddc0f8f2f0a73b9fa0d48e4b2b6e/haproxy/0.log"
Sep 16 19:37:46.336518 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.336489    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kzxkd/must-gather-8lhk9"]
Sep 16 19:37:46.336936 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.336693    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-kzxkd/must-gather-8lhk9" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="copy" containerID="cri-o://a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b" gracePeriod=2
Sep 16 19:37:46.341445 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.341422    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kzxkd/must-gather-8lhk9"]
Sep 16 19:37:46.564565 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.564544    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kzxkd_must-gather-8lhk9_3420e696-2017-4746-b766-a8655e2c7e27/copy/0.log"
Sep 16 19:37:46.566098 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.566081    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:46.568102 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:37:46.568017    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8lhk9\" is forbidden: User \"system:node:ip-10-0-128-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzxkd\": no relationship found between node 'ip-10-0-128-180.ec2.internal' and this object" podUID="3420e696-2017-4746-b766-a8655e2c7e27" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:46.707005 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.706922    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txzmv\" (UniqueName: \"kubernetes.io/projected/3420e696-2017-4746-b766-a8655e2c7e27-kube-api-access-txzmv\") pod \"3420e696-2017-4746-b766-a8655e2c7e27\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") "
Sep 16 19:37:46.707152 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.707042    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3420e696-2017-4746-b766-a8655e2c7e27-must-gather-output\") pod \"3420e696-2017-4746-b766-a8655e2c7e27\" (UID: \"3420e696-2017-4746-b766-a8655e2c7e27\") "
Sep 16 19:37:46.708167 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.708138    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3420e696-2017-4746-b766-a8655e2c7e27-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3420e696-2017-4746-b766-a8655e2c7e27" (UID: "3420e696-2017-4746-b766-a8655e2c7e27"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:46.709386 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.709358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3420e696-2017-4746-b766-a8655e2c7e27-kube-api-access-txzmv" (OuterVolumeSpecName: "kube-api-access-txzmv") pod "3420e696-2017-4746-b766-a8655e2c7e27" (UID: "3420e696-2017-4746-b766-a8655e2c7e27"). InnerVolumeSpecName "kube-api-access-txzmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:46.807855 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.807818    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3420e696-2017-4746-b766-a8655e2c7e27-must-gather-output\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:37:46.807855 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.807856    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzmv\" (UniqueName: \"kubernetes.io/projected/3420e696-2017-4746-b766-a8655e2c7e27-kube-api-access-txzmv\") on node \"ip-10-0-128-180.ec2.internal\" DevicePath \"\""
Sep 16 19:37:46.970474 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.970393    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kzxkd_must-gather-8lhk9_3420e696-2017-4746-b766-a8655e2c7e27/copy/0.log"
Sep 16 19:37:46.970782 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.970757    2562 generic.go:356] "Generic (PLEG): container finished" podID="3420e696-2017-4746-b766-a8655e2c7e27" containerID="a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b" exitCode=143
Sep 16 19:37:46.970867 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.970808    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:46.970867 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.970851    2562 scope.go:117] "RemoveContainer" containerID="a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b"
Sep 16 19:37:46.972751 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:37:46.972721    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8lhk9\" is forbidden: User \"system:node:ip-10-0-128-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzxkd\": no relationship found between node 'ip-10-0-128-180.ec2.internal' and this object" podUID="3420e696-2017-4746-b766-a8655e2c7e27" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:46.979242 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.979224    2562 scope.go:117] "RemoveContainer" containerID="e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"
Sep 16 19:37:46.981014 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:37:46.980986    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8lhk9\" is forbidden: User \"system:node:ip-10-0-128-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzxkd\": no relationship found between node 'ip-10-0-128-180.ec2.internal' and this object" podUID="3420e696-2017-4746-b766-a8655e2c7e27" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:46.992011 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.991994    2562 scope.go:117] "RemoveContainer" containerID="a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b"
Sep 16 19:37:46.992297 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:37:46.992281    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b\": container with ID starting with a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b not found: ID does not exist" containerID="a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b"
Sep 16 19:37:46.992392 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.992309    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b"} err="failed to get container status \"a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b\": rpc error: code = NotFound desc = could not find container \"a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b\": container with ID starting with a1889416b2271620102ca04d4e9a83bb8eed6e8c82960af512c6627851fac28b not found: ID does not exist"
Sep 16 19:37:46.992392 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.992358    2562 scope.go:117] "RemoveContainer" containerID="e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"
Sep 16 19:37:46.992626 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:37:46.992604    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9\": container with ID starting with e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9 not found: ID does not exist" containerID="e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"
Sep 16 19:37:46.992720 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:46.992628    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9"} err="failed to get container status \"e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9\": rpc error: code = NotFound desc = could not find container \"e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9\": container with ID starting with e4af7f836c936b150574adf344197ca83aba878ff984eb204b491f86d4a850c9 not found: ID does not exist"
Sep 16 19:37:47.372731 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:47.372645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:47.467835 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:47.467791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2thsb_b4d6f9b7-eb42-43f6-a88e-70a0f332226b/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:48.120880 ip-10-0-128-180 kubenswrapper[2562]: E0916 19:37:48.120839    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8lhk9\" is forbidden: User \"system:node:ip-10-0-128-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzxkd\": no relationship found between node 'ip-10-0-128-180.ec2.internal' and this object" podUID="3420e696-2017-4746-b766-a8655e2c7e27" pod="openshift-must-gather-kzxkd/must-gather-8lhk9"
Sep 16 19:37:48.126344 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:48.123515    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3420e696-2017-4746-b766-a8655e2c7e27" path="/var/lib/kubelet/pods/3420e696-2017-4746-b766-a8655e2c7e27/volumes"
Sep 16 19:37:48.977149 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:48.977056    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/alertmanager/0.log"
Sep 16 19:37:49.006968 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.006940    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/config-reloader/0.log"
Sep 16 19:37:49.035362 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.035335    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/kube-rbac-proxy-web/0.log"
Sep 16 19:37:49.063878 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.063853    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/kube-rbac-proxy/0.log"
Sep 16 19:37:49.089865 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.089842    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/kube-rbac-proxy-metric/0.log"
Sep 16 19:37:49.113399 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.113375    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/prom-label-proxy/0.log"
Sep 16 19:37:49.139211 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.139195    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e30a1f74-7125-4828-a88f-1ad184201b9b/init-config-reloader/0.log"
Sep 16 19:37:49.225371 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.225343    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pdwkp_53da9125-8892-4a1b-9bd8-c5974d12cf52/kube-state-metrics/0.log"
Sep 16 19:37:49.248951 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.248856    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pdwkp_53da9125-8892-4a1b-9bd8-c5974d12cf52/kube-rbac-proxy-main/0.log"
Sep 16 19:37:49.280291 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.280272    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-pdwkp_53da9125-8892-4a1b-9bd8-c5974d12cf52/kube-rbac-proxy-self/0.log"
Sep 16 19:37:49.319016 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.318993    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bf57fd676-rx9x8_fc57feab-c598-46a1-8d33-2ca94832f77f/metrics-server/0.log"
Sep 16 19:37:49.560527 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.560493    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rtv7r_0ea21f56-d2d5-41af-89eb-4b370def1c75/node-exporter/0.log"
Sep 16 19:37:49.586501 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.586476    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rtv7r_0ea21f56-d2d5-41af-89eb-4b370def1c75/kube-rbac-proxy/0.log"
Sep 16 19:37:49.613073 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.613041    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rtv7r_0ea21f56-d2d5-41af-89eb-4b370def1c75/init-textfile/0.log"
Sep 16 19:37:49.642572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.642538    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mqknv_100ace8a-d8dc-45ae-906e-c72d0cdca137/kube-rbac-proxy-main/0.log"
Sep 16 19:37:49.671446 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.671425    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mqknv_100ace8a-d8dc-45ae-906e-c72d0cdca137/kube-rbac-proxy-self/0.log"
Sep 16 19:37:49.699347 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.699304    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mqknv_100ace8a-d8dc-45ae-906e-c72d0cdca137/openshift-state-metrics/0.log"
Sep 16 19:37:49.754000 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.753947    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/prometheus/0.log"
Sep 16 19:37:49.778010 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.777979    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/config-reloader/0.log"
Sep 16 19:37:49.803946 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.803919    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/thanos-sidecar/0.log"
Sep 16 19:37:49.829560 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.829488    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/kube-rbac-proxy-web/0.log"
Sep 16 19:37:49.856252 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.856224    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/kube-rbac-proxy/0.log"
Sep 16 19:37:49.881377 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.881354    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/kube-rbac-proxy-thanos/0.log"
Sep 16 19:37:49.909032 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:49.909007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bac1f1e1-c420-4f63-a353-5060a5ccbcbc/init-config-reloader/0.log"
Sep 16 19:37:50.030572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:50.030545    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fbfc7d557-jzvtg_e3bafb0d-b88a-484d-9500-e5514255974a/telemeter-client/0.log"
Sep 16 19:37:50.053976 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:50.053947    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fbfc7d557-jzvtg_e3bafb0d-b88a-484d-9500-e5514255974a/reload/0.log"
Sep 16 19:37:50.078425 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:50.078388    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fbfc7d557-jzvtg_e3bafb0d-b88a-484d-9500-e5514255974a/kube-rbac-proxy/0.log"
Sep 16 19:37:51.533953 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:51.533928    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-29v4m_e4f2a7cb-f471-4ef9-b4fb-40535d0d5a57/networking-console-plugin/0.log"
Sep 16 19:37:52.709971 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.709933    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"]
Sep 16 19:37:52.710365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.710266    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="copy"
Sep 16 19:37:52.710365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.710278    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="copy"
Sep 16 19:37:52.710365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.710290    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="gather"
Sep 16 19:37:52.710365 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.710295    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="gather"
Sep 16 19:37:52.710497 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.710373    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="gather"
Sep 16 19:37:52.710497 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.710384    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3420e696-2017-4746-b766-a8655e2c7e27" containerName="copy"
Sep 16 19:37:52.713959 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.713939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.716496 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.716475    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rwfh7\"/\"default-dockercfg-wx2v9\""
Sep 16 19:37:52.716623 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.716481    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rwfh7\"/\"openshift-service-ca.crt\""
Sep 16 19:37:52.718014 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.717994    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rwfh7\"/\"kube-root-ca.crt\""
Sep 16 19:37:52.722332 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.722293    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"]
Sep 16 19:37:52.762941 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.762912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-sys\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.763055 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.762950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-podres\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.763055 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.762972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7rc\" (UniqueName: \"kubernetes.io/projected/daee490f-b544-43bb-900c-5c139e1854c6-kube-api-access-6h7rc\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.763055 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.763020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-proc\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.763055 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.763041    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-lib-modules\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863806 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-lib-modules\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-sys\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-podres\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7rc\" (UniqueName: \"kubernetes.io/projected/daee490f-b544-43bb-900c-5c139e1854c6-kube-api-access-6h7rc\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-sys\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-proc\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-lib-modules\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.863986 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-podres\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.864273 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.863997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/daee490f-b544-43bb-900c-5c139e1854c6-proc\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:52.873022 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:52.873000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7rc\" (UniqueName: \"kubernetes.io/projected/daee490f-b544-43bb-900c-5c139e1854c6-kube-api-access-6h7rc\") pod \"perf-node-gather-daemonset-t45m4\" (UID: \"daee490f-b544-43bb-900c-5c139e1854c6\") " pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:53.024507 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:53.024414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:53.294828 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:53.294750    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"]
Sep 16 19:37:53.297017 ip-10-0-128-180 kubenswrapper[2562]: W0916 19:37:53.296992    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddaee490f_b544_43bb_900c_5c139e1854c6.slice/crio-97c51a68d92ef8cdf5bfcdf1535bb40f742f56df9fc030cd1333aa48205392c4 WatchSource:0}: Error finding container 97c51a68d92ef8cdf5bfcdf1535bb40f742f56df9fc030cd1333aa48205392c4: Status 404 returned error can't find the container with id 97c51a68d92ef8cdf5bfcdf1535bb40f742f56df9fc030cd1333aa48205392c4
Sep 16 19:37:54.006207 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.006172    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4" event={"ID":"daee490f-b544-43bb-900c-5c139e1854c6","Type":"ContainerStarted","Data":"1efeabe48fe709e54a6e560168dd76ff087928969ebf191ff5c57bfa4a349d9e"}
Sep 16 19:37:54.006607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.006228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4" event={"ID":"daee490f-b544-43bb-900c-5c139e1854c6","Type":"ContainerStarted","Data":"97c51a68d92ef8cdf5bfcdf1535bb40f742f56df9fc030cd1333aa48205392c4"}
Sep 16 19:37:54.006607 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.006255    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:37:54.022494 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.022445    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4" podStartSLOduration=2.022407633 podStartE2EDuration="2.022407633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:54.021777762 +0000 UTC m=+3276.438480522" watchObservedRunningTime="2026-09-16 19:37:54.022407633 +0000 UTC m=+3276.439110429"
Sep 16 19:37:54.076672 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.076639    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cfxqx_f2740806-51d2-4432-a42d-384de809b042/dns/0.log"
Sep 16 19:37:54.099210 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.099164    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cfxqx_f2740806-51d2-4432-a42d-384de809b042/kube-rbac-proxy/0.log"
Sep 16 19:37:54.204477 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.204447    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmdz7_3f93bc5c-ba37-4cbc-8c8e-9062254a738c/dns-node-resolver/0.log"
Sep 16 19:37:54.767726 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:54.767687    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4m88x_572231be-7380-44a9-8379-caf0f42c820d/node-ca/0.log"
Sep 16 19:37:56.061213 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:56.061185    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq67w_80e76374-0e57-4816-9207-e5837032f7c1/serve-healthcheck-canary/0.log"
Sep 16 19:37:56.451729 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:56.451632    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gn5r4_a180ce87-cf24-4c64-bbc8-0dd2ae85013f/insights-operator/0.log"
Sep 16 19:37:56.453036 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:56.453001    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gn5r4_a180ce87-cf24-4c64-bbc8-0dd2ae85013f/insights-operator/1.log"
Sep 16 19:37:56.551644 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:56.551617    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4jcd_a181ba71-9d0f-4556-8a94-f7ee21ed9fc3/kube-rbac-proxy/0.log"
Sep 16 19:37:56.573162 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:56.573140    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4jcd_a181ba71-9d0f-4556-8a94-f7ee21ed9fc3/exporter/0.log"
Sep 16 19:37:56.595972 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:56.595935    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4jcd_a181ba71-9d0f-4556-8a94-f7ee21ed9fc3/extractor/0.log"
Sep 16 19:37:59.694278 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:59.694234    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cgr4p_152dcbb0-4b84-4852-8102-c09114d9fe39/s3-init/0.log"
Sep 16 19:37:59.717250 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:59.717219    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cgr4p_152dcbb0-4b84-4852-8102-c09114d9fe39/download-hf-model/0.log"
Sep 16 19:37:59.759039 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:59.759005    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-nnjzs_d9e7d6e1-e949-4d54-999f-0326cba10469/s3-tls-init-custom/0.log"
Sep 16 19:37:59.777572 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:59.777555    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-nnjzs_d9e7d6e1-e949-4d54-999f-0326cba10469/download-hf-model/0.log"
Sep 16 19:37:59.812768 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:59.812743    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-fl5tx_07c7cc52-6837-4c21-84e4-0cc10af90bae/s3-tls-init-serving/0.log"
Sep 16 19:37:59.857689 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:37:59.857669    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-fl5tx_07c7cc52-6837-4c21-84e4-0cc10af90bae/download-hf-model/0.log"
Sep 16 19:38:00.021076 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:00.021000    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rwfh7/perf-node-gather-daemonset-t45m4"
Sep 16 19:38:05.766336 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:05.766285    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-22sjk_64cea152-19dd-4c44-9119-5d00570bd11f/kube-multus/0.log"
Sep 16 19:38:05.968444 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:05.968415    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:38:05.993313 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:05.993281    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/egress-router-binary-copy/0.log"
Sep 16 19:38:06.019085 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.019001    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/cni-plugins/0.log"
Sep 16 19:38:06.043358 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.043314    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/bond-cni-plugin/0.log"
Sep 16 19:38:06.069028 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.068990    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/routeoverride-cni/0.log"
Sep 16 19:38:06.092601 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.092567    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/whereabouts-cni-bincopy/0.log"
Sep 16 19:38:06.117457 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.117423    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mm2fq_3c9ced61-ba03-421f-bda1-20977b70864c/whereabouts-cni/0.log"
Sep 16 19:38:06.455809 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.455768    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5ps6t_2cb5a23e-f4e8-4542-b175-5fe7be200604/network-metrics-daemon/0.log"
Sep 16 19:38:06.478512 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:06.478486    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5ps6t_2cb5a23e-f4e8-4542-b175-5fe7be200604/kube-rbac-proxy/0.log"
Sep 16 19:38:07.538457 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.538421    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/ovn-controller/0.log"
Sep 16 19:38:07.557180 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.557142    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/ovn-acl-logging/0.log"
Sep 16 19:38:07.583907 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.583877    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/kube-rbac-proxy-node/0.log"
Sep 16 19:38:07.606825 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.606802    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:38:07.625868 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.625842    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/northd/0.log"
Sep 16 19:38:07.647347 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.647309    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/nbdb/0.log"
Sep 16 19:38:07.669994 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.669974    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/sbdb/0.log"
Sep 16 19:38:07.736856 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:07.736832    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lzd4j_8c6d19d5-9dce-406f-a363-e4126893ef82/ovnkube-controller/0.log"
Sep 16 19:38:09.035348 ip-10-0-128-180 kubenswrapper[2562]: I0916 19:38:09.034762    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jgp54_dd75c1a3-b9c4-4300-928c-efe3b8c85ee4/network-check-target-container/0.log"