Sep 17 17:34:14.070744 ip-10-0-133-71 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:17.504310 ip-10-0-133-71 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:17.504310 ip-10-0-133-71 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:17.504310 ip-10-0-133-71 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:17.504310 ip-10-0-133-71 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:17.504310 ip-10-0-133-71 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:17.505136 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.504702    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510093    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510116    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510123    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510127    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510131    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510135    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:17.510130 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510140    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510146    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510150    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510154    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510158    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510162    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510167    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510171    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510175    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510179    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510183    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510187    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510192    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510196    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510200    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510204    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510208    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510214    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510220    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:17.510573 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510225    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510230    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510234    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510239    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510245    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510251    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510255    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510266    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510271    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510276    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510280    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510285    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510289    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510294    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510299    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510305    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510310    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510314    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510319    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510324    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:17.511391 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510328    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510333    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510340    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510346    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510351    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510356    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510364    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510370    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510375    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510380    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510385    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510390    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510394    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510399    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510419    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510424    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510428    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510433    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510437    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510442    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:17.512438 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510446    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510451    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510455    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510460    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510465    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510470    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510475    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510482    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510487    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510492    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510497    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510502    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510506    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510511    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510515    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510522    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510526    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510531    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510537    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510542    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:17.513397 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510547    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510552    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510557    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510561    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510566    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510570    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510575    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510579    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510583    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510587    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510592    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510596    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510601    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.510606    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511902    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511913    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511919    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511927    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511933    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:17.514338 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511941    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511946    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511952    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511956    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511961    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511966    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511972    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511978    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511982    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511987    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511991    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.511996    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512001    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512005    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512009    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512014    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512017    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512022    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512026    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512030    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:17.515246 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512035    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512039    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512045    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512050    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512054    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512058    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512063    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512068    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512072    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512077    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512081    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512087    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512093    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512098    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512102    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512107    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512111    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512119    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512125    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512130    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:17.515947 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512135    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512140    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512145    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512150    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512155    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512161    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512166    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512171    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512177    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512181    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512186    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512190    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512194    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512199    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512204    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512209    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512213    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512218    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512223    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512227    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:17.516704 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512232    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512236    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512240    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512245    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512250    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512255    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512260    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512264    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512268    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512275    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512279    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512283    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512288    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512293    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512298    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512302    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512307    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512312    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512324    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512329    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:17.517294 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512334    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512338    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512343    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512348    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512352    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512357    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512363    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512369    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512375    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512380    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512385    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512389    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512394    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.512398    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513107    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513119    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513126    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513131    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513136    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:17.518018 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513141    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513147    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513152    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513156    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513161    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513166    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513170    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513175    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513179    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513184    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513188    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513194    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513199    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513206    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513213    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513218    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513223    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513228    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513232    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513237    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:17.518558 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513241    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513245    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513250    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513254    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513258    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513263    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513269    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513274    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513279    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513284    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513288    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513293    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513297    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513302    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513306    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513311    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513315    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513320    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513325    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513329    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:17.519180 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513334    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513339    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513345    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513350    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513367    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513372    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513377    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513382    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513386    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513391    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513395    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513399    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513423    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513427    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513431    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513436    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513440    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513445    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513449    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513454    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:17.519737 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513458    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513464    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513468    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513475    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513482    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513487    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513491    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513496    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513500    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513505    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513509    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513522    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513528    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513532    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513537    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513541    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513548    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513563    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:17.520261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513573    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513578    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513582    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513587    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513591    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513596    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513601    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513605    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513610    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513615    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513620    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513624    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513629    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513633    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513638    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.513643    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513753    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513809    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513820    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513835    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513842    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:17.520773 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513847    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513854    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513862    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513867    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513872    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513878    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513884    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513889    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513894    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513899    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513904    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513925    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513931    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513943    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513948    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513953    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513959    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513965    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513972    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513977    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513984    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513989    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.513995    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514000    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:17.521318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514005    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514012    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514019    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514026    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514032    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514037    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514042    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514047    2562 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514053    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514060    2562 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514065    2562 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514070    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514075    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514080    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514086    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514091    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514096    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514102    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514107    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514114    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514120    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514125    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514130    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514136    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514140    2562 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:17.522029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514147    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514153    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514158    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514163    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514168    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514174    2562 flags.go:64] FLAG: --help="false"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514179    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514184    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514189    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514196    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514203    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514208    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514214    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514219    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514224    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514229    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514235    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514240    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514245    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514250    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514255    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514260    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514265    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514270    2562 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:17.522760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514275    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514280    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514290    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514299    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514306    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514311    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514316    2562 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514322    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514327    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514332    2562 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514337    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514344    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514349    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514355    2562 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514360    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514364    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514369    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514376    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514381    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514386    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514391    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514425    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514430    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514436    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:17.523398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514441    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514445    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514455    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514460    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514466    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514471    2562 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514477    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514482    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-096c41ec6edf37281"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514488    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514493    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514501    2562 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514507    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514513    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514519    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514525    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514530    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514536    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514541    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514547    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514552    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514557    2562 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514562    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514567    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514573    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514578    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:17.524062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514582    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514591    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514596    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514601    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514607    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514612    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514617    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514622    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514627    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514632    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514637    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514647    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514653    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514658    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514665    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514670    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514675    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514682    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514687    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514692    2562 flags.go:64] FLAG: --v="2"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514699    2562 flags.go:64] FLAG: --version="false"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514712    2562 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514718    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.514724    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514931    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:17.524747 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514939    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514944    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514950    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514955    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514960    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514965    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514970    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514974    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514981    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514986    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514991    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.514996    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515000    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515004    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515009    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515013    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515018    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515022    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515026    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515031    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:17.525448 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515035    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515041    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515045    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515050    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515056    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515060    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515065    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515069    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515074    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515080    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515085    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515089    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515093    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515098    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515103    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515107    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515111    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515117    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515121    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:17.526009 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515125    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515132    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515137    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515141    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515146    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515150    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515154    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515159    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515163    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515168    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515174    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515181    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515186    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515191    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515196    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515200    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515205    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515213    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515218    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515222    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:17.526550 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515227    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515231    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515236    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515241    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515245    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515249    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515253    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515257    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515262    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515267    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515271    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515276    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515280    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515286    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515291    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515296    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515300    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515304    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515309    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515313    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:17.527125 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515318    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515322    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515327    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515331    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515336    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515340    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515345    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515349    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515354    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515363    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515369    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515374    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515381    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515387    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515392    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515399    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515424    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515429    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:17.527682 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.515434    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.516049    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.523617    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.523635    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.523657    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.523667    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523727    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523732    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523736    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523739    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523742    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523746    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523749    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523752    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523755    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:17.528187 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523758    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523761    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523763    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523768    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523773    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523776    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523779    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523782    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523785    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523788    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523791    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523794    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523796    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523799    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523802    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523805    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523808    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523810    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523813    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:17.528756 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523816    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523820    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523824    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523826    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523829    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523832    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523835    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523837    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523840    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523843    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523846    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523848    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523851    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523854    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523857    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523860    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523863    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523865    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523868    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523871    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:17.529268 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523873    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523876    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523879    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523881    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523884    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523887    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523890    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523893    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523895    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523898    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523900    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523903    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523906    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523909    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523912    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523915    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523919    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523922    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523924    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523927    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:17.529808 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523930    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523933    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523936    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523939    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523942    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523944    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523947    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523950    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523954    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523958    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523961    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523964    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523967    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523970    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523974    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523978    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523981    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523984    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523987    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:17.530343 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523989    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523992    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523995    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.523998    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524001    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524004    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524008    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524011    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524015    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524018    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524020    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524023    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.524028    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524153    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524158    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:17.530877 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524162    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524165    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524168    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524172    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524175    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524178    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524181    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524183    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524186    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524189    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524192    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524195    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524198    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524200    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524203    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524206    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524209    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524211    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524214    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524217    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:17.531307 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524220    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524224    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524228    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524231    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524235    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524239    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524242    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524246    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524249    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524252    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524255    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524258    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524261    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524265    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524268    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524271    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524273    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524276    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524279    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524282    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:17.531865 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524285    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524287    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524290    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524293    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524296    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524299    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524302    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524304    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524307    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524310    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524313    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524316    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524318    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524321    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524324    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524328    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524332    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524336    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524339    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:17.532663 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524343    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524346    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524349    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524352    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524355    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524358    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524360    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524363    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524366    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524369    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524372    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524374    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524377    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524380    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524383    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524386    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524389    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524391    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524394    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524397    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:17.533347 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524400    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524416    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524419    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524422    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524425    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524428    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524431    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524433    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524437    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524440    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524445    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524448    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524451    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524455    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524458    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524461    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524464    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:17.534137 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:17.524467    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.524472    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.524588    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.526581    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.527347    2562 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.527467    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.527505    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.529674    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:17.534712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.531007    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:17.541057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.541033    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:17.544595 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.544578    2562 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:17.547030 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.547006    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:17.550165 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.550145    2562 fs.go:135] Filesystem UUIDs: map[1daca5ee-d431-4264-ae7b-f2a85248d04c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 92b39271-b603-407b-a456-08bdbfa7db22:/dev/nvme0n1p4]
Sep 17 17:34:17.550241 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.550166    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:17.554899 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.554777    2562 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:17.554213393 +0000 UTC m=+2.002043865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099147 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a0e23b55da80bb8237795db1fb2c7 SystemUUID:ec2a0e23-b55d-a80b-b823-7795db1fb2c7 BootID:51769226-5445-499e-8ee4-3a3b7faf64fe Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:2c:59:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:2c:59:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:2b:93:6a:41:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:17.554899 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.554895    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:17.555015 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.554988    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:17.556529 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.556494    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:17.556692 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.556529    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:17.557610 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.557598    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:17.557654 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.557612    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:17.557654 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.557621    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:17.557654 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.557633    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:17.557740 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.557714    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:17.561578 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561565    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:17.561745 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561735    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:17.561785 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561775    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:17.561816 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561794    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:17.561854 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561824    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:17.561854 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561840    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:17.561854 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.561853    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:17.562361 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.562348    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:17.562444 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.562370    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:17.564457 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.564435    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:17.566867 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.566852    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:17.566950 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.566885    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:17.568339 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568326    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568344    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568351    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568357    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568367    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568373    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568388    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568394    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:17.568417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568417    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:17.568645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568423    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:17.568645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568433    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:17.568645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568440    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:17.568645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568470    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:17.568645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.568477    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:17.572444 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.572430    2562 server.go:1268] "Started kubelet"
Sep 17 17:34:17.572569 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.572541    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:17.572630 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.572518    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:17.572683 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.572626    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:17.573468 ip-10-0-133-71 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:17.574008 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.573993    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:17.577362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.577342    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:17.605030 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.605002    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:17.605141 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.605013    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:17.605141 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.604913    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:17.605141 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.605115    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:17.605477 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.605451    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:17.610910 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.610020    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c293e08a729  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.572386601 +0000 UTC m=+2.020217049,LastTimestamp:2026-09-17 17:34:17.572386601 +0000 UTC m=+2.020217049,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.611022 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.610916    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:17.611022 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.610916    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:17.611133 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.611066    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:17.611536 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611511    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:17.611646 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611518    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:17.611646 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611555    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:17.611772 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611676    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:17.611772 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611689    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:17.611920 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611903    2562 factory.go:55] Registering systemd factory
Sep 17 17:34:17.611979 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.611924    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:17.612162 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.612143    2562 factory.go:153] Registering CRI-O factory
Sep 17 17:34:17.612249 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.612168    2562 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:17.612249 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.612222    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:17.612249 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.612241    2562 factory.go:103] Registering Raw factory
Sep 17 17:34:17.612375 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.612255    2562 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:17.612745 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.612730    2562 manager.go:349] Starting recovery of all containers
Sep 17 17:34:17.623837 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.623820    2562 manager.go:354] Recovery completed
Sep 17 17:34:17.624877 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.624849    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:34:17.624953 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.624878    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:34:17.628048 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.628035    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.630129 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630100    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.630229 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630146    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.630229 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630163    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.630780 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630766    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:17.630780 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630772    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.630780 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630777    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:17.630939 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.630793    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:17.631535 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.631517    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.631602 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.631553    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.631602 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.631569    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.633090 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.633073    2562 policy_none.go:49] "None policy: Start"
Sep 17 17:34:17.633173 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.633106    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:17.633173 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.633118    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:17.635234 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.635221    2562 policy_none.go:47] "Start"
Sep 17 17:34:17.648325 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.648230    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c294179b7b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630128052 +0000 UTC m=+2.077958509,LastTimestamp:2026-09-17 17:34:17.630128052 +0000 UTC m=+2.077958509,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.655567 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.655499    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c29417a1ae3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630153443 +0000 UTC m=+2.077983893,LastTimestamp:2026-09-17 17:34:17.630153443 +0000 UTC m=+2.077983893,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.666499 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.665977    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c29417a5373  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630167923 +0000 UTC m=+2.077998372,LastTimestamp:2026-09-17 17:34:17.630167923 +0000 UTC m=+2.077998372,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671029    2562 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.671079    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671093    2562 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671381    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671401    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671511    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671596    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.671606    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.672098    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.672143    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:17.686067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.685732    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d62c294179b7b4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c294179b7b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630128052 +0000 UTC m=+2.077958509,LastTimestamp:2026-09-17 17:34:17.631533992 +0000 UTC m=+2.079364444,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.701878 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.701799    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d62c29417a1ae3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c29417a1ae3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630153443 +0000 UTC m=+2.077983893,LastTimestamp:2026-09-17 17:34:17.631560543 +0000 UTC m=+2.079390995,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.712213 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.712141    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d62c29417a5373\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c29417a5373  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630167923 +0000 UTC m=+2.077998372,LastTimestamp:2026-09-17 17:34:17.631575502 +0000 UTC m=+2.079405953,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.719776 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.719748    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:17.721091 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.721071    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:17.721192 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.721127    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:17.721192 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.721151    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:17.721274 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.721196    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:17.724534 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.724504    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 17:34:17.724639 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.724575    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c2944cc9e1e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.685892638 +0000 UTC m=+2.133723083,LastTimestamp:2026-09-17 17:34:17.685892638 +0000 UTC m=+2.133723083,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.766247 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.766170    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ds2m9"
Sep 17 17:34:17.772361 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.772326    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.773372 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.773353    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.773463 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.773384    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.773463 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.773397    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.773463 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.773428    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.786567 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.786483    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d62c294179b7b4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c294179b7b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630128052 +0000 UTC m=+2.077958509,LastTimestamp:2026-09-17 17:34:17.773369779 +0000 UTC m=+2.221200228,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.787036 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.787014    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ds2m9"
Sep 17 17:34:17.789374 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.789297    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-71.ec2.internal.18d62c29417a1ae3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-71.ec2.internal.18d62c29417a1ae3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-71.ec2.internal,UID:ip-10-0-133-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-71.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-71.ec2.internal,},FirstTimestamp:2026-09-17 17:34:17.630153443 +0000 UTC m=+2.077983893,LastTimestamp:2026-09-17 17:34:17.773389544 +0000 UTC m=+2.221219993,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-71.ec2.internal,}"
Sep 17 17:34:17.789601 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.789579    2562 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.821645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.821611    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"]
Sep 17 17:34:17.821783 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.821689    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.822713 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.822694    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.822805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.822731    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.822805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.822746    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.825234 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.825215    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.825390 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.825370    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.825465 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.825412    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.826098 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.826083    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.826169 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.826098    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.826169 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.826116    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.826169 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.826134    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.826297 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.826119    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.826297 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.826278    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.828612 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.828594    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.828703 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.828630    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.829459 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.829439    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.829554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.829472    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.829554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.829486    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.841481 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.841462    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-71.ec2.internal\" not found" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.857174 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.857154    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-71.ec2.internal\" not found" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.861853 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:17.861837    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-71.ec2.internal\" not found" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.913166 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.913139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4abd5f1ae2c9430a917972c6d312f06a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-71.ec2.internal\" (UID: \"4abd5f1ae2c9430a917972c6d312f06a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.913166 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.913167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4af3ae50a7b4cbff8bf701106a65509-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"b4af3ae50a7b4cbff8bf701106a65509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.913384 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.913187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4af3ae50a7b4cbff8bf701106a65509-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"b4af3ae50a7b4cbff8bf701106a65509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:17.991215 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.991182    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:17.992283 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.992265    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:17.992397 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.992297    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:17.992397 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.992310    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:17.992397 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:17.992356    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.006465 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.006445    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.006525 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.006475    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-71.ec2.internal\": node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.013623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.013605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4af3ae50a7b4cbff8bf701106a65509-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"b4af3ae50a7b4cbff8bf701106a65509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.013691 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.013634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4af3ae50a7b4cbff8bf701106a65509-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"b4af3ae50a7b4cbff8bf701106a65509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.013691 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.013654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4abd5f1ae2c9430a917972c6d312f06a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-71.ec2.internal\" (UID: \"4abd5f1ae2c9430a917972c6d312f06a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.013766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.013697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4abd5f1ae2c9430a917972c6d312f06a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-71.ec2.internal\" (UID: \"4abd5f1ae2c9430a917972c6d312f06a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.013766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.013720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4af3ae50a7b4cbff8bf701106a65509-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"b4af3ae50a7b4cbff8bf701106a65509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.013766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.013721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4af3ae50a7b4cbff8bf701106a65509-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal\" (UID: \"b4af3ae50a7b4cbff8bf701106a65509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.050013 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.049925    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.152006 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.151958    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.160839 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.160816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.165064 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.165028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:18.253030 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.252980    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.353367 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.353099    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.454520 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.454459    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.528038 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.527991    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:18.556451 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.556398    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.591666 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.591612    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:18.656636 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.656552    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.758098 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.758062    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.792392 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.792322    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:18 +0000 UTC" deadline="2028-06-03 17:29:21.85100156 +0000 UTC"
Sep 17 17:34:18.792392 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.792391    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14999h55m3.058614899s"
Sep 17 17:34:18.793493 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.793465    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:18.794738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.794717    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:18.794821 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.794750    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:18.794821 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.794762    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:18.859168 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.859129    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:18.864002 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:18.863962    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abd5f1ae2c9430a917972c6d312f06a.slice/crio-9628a037220a6a4bc8738b51d05dea8b951a6588ba423cdd91807a2e7942ec7d WatchSource:0}: Error finding container 9628a037220a6a4bc8738b51d05dea8b951a6588ba423cdd91807a2e7942ec7d: Status 404 returned error can't find the container with id 9628a037220a6a4bc8738b51d05dea8b951a6588ba423cdd91807a2e7942ec7d
Sep 17 17:34:18.864250 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:18.864232    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4af3ae50a7b4cbff8bf701106a65509.slice/crio-2a2982e04b26ae8eac3fd65d44cab0ad53985a2495ba5b2ac92863e1f3542c6e WatchSource:0}: Error finding container 2a2982e04b26ae8eac3fd65d44cab0ad53985a2495ba5b2ac92863e1f3542c6e: Status 404 returned error can't find the container with id 2a2982e04b26ae8eac3fd65d44cab0ad53985a2495ba5b2ac92863e1f3542c6e
Sep 17 17:34:18.869283 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:18.869262    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:18.960150 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:18.960045    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-71.ec2.internal\" not found"
Sep 17 17:34:19.023346 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.023314    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:19.085756 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.085719    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:19.109424 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.109400    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:19.111448 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.111427    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:19.139595 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.139562    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:19.140610 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.140591    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal"
Sep 17 17:34:19.162041 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.162013    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:19.562935 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.562895    2562 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:19.573724 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.573684    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:19.574123 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.574098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wzs8n","openshift-multus/multus-fz8ls","openshift-multus/network-metrics-daemon-rzpqd","openshift-network-diagnostics/network-check-target-7c8nf","kube-system/global-pull-secret-syncer-d2v8l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq","openshift-cluster-node-tuning-operator/tuned-qpq7l","openshift-image-registry/node-ca-j9mf9","openshift-network-operator/iptables-alerter-b9dts","openshift-ovn-kubernetes/ovnkube-node-5l8m4","kube-system/konnectivity-agent-wb25w","kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal"]
Sep 17 17:34:19.577259 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.577003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.579556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.579533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.582402 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.582344    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:19.582504 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.582458    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:19.585081 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.585060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:19.585219 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.585127    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:19.586492 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.585883    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.586492 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.586041    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6n86b\""
Sep 17 17:34:19.586492 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.586095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:19.587359 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.587337    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:19.587476 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.587418    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:19.587669 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.587624    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.587669 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.587657    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:19.587793 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.587757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.587836 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.587820    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:19.589195 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.588991    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpbpx\""
Sep 17 17:34:19.589973 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.589953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.592999 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.592451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.595272 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.594891    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.597639 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.597518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.597786 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.597758    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.598123 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.598096    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.598227 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.598102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:19.599112 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.599089    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dw7vm\""
Sep 17 17:34:19.600324 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.600252    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.602794 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.602773    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.606558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.606537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.606661 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.606564    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7ws8d\""
Sep 17 17:34:19.606951 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.606930    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.607042 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.607022    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.607290 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.607271    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:19.607364 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.607293    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.607364 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.607343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.607582 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.607565    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:19.607647 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.607609    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:19.608879 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.608856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:19.609431 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.609287    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:19.609431 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.609343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:19.609431 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.609350    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4r7lt\""
Sep 17 17:34:19.611439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.610068    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r2m68\""
Sep 17 17:34:19.611439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.610218    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:19.611439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.610663    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:19.611439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.611258    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:19.611439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.611310    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8qj5\""
Sep 17 17:34:19.611714 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.611610    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.612365 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.611998    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:19.612365 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.612066    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:19.612523 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.612484    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-px5tz\""
Sep 17 17:34:19.613604 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.613391    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:19.623622 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.623738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-registration-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.623738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.623738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.623914 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-k8s-cni-cncf-io\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.623914 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-hostroot\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.623914 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589b2\" (UniqueName: \"kubernetes.io/projected/3345f800-626d-4db5-b7b5-38d77107cc0b-kube-api-access-589b2\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-host\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.623978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-var-lib-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-ovn\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-modprobe-d\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624190    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9557f36-6cf8-46fb-b997-9f9ade8915be-host\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p7h\" (UniqueName: \"kubernetes.io/projected/a5c1752e-456c-417e-9ef1-25e6efc078d1-kube-api-access-45p7h\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mftd\" (UniqueName: \"kubernetes.io/projected/1e806d7b-ef85-4912-912f-428b60ebc744-kube-api-access-4mftd\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-env-overrides\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cnibin\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9557f36-6cf8-46fb-b997-9f9ade8915be-serviceca\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74b6bb2c-1e44-46bd-90c8-e6c1772d35f9-agent-certs\") pod \"konnectivity-agent-wb25w\" (UID: \"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9\") " pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-device-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-log-socket\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovnkube-script-lib\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.624664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysctl-conf\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-var-lib-kubelet\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-cni-bin\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-cni-multus\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e806d7b-ef85-4912-912f-428b60ebc744-multus-daemon-config\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-kubernetes\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpglz\" (UniqueName: \"kubernetes.io/projected/59ba2826-88d8-472b-8c1d-2f9d06841f6d-kube-api-access-gpglz\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c1752e-456c-417e-9ef1-25e6efc078d1-iptables-alerter-script\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-socket-dir-parent\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-sys-fs\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.624997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4fb\" (UniqueName: \"kubernetes.io/projected/aa0b68f5-956e-4884-8f76-7ee1920966e1-kube-api-access-rv4fb\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-run-ovn-kubernetes\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c1752e-456c-417e-9ef1-25e6efc078d1-host-slash\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-netns\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngb75\" (UniqueName: \"kubernetes.io/projected/fbc60926-841c-4824-872b-30a7c806ade0-kube-api-access-ngb75\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:19.625225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysconfig\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e806d7b-ef85-4912-912f-428b60ebc744-cni-binary-copy\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-dbus\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-etc-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-cni-netd\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovnkube-config\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovn-node-metrics-cert\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74b6bb2c-1e44-46bd-90c8-e6c1772d35f9-konnectivity-ca\") pod \"konnectivity-agent-wb25w\" (UID: \"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9\") " pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-sys\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-systemd\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-systemd\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-lib-modules\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-systemd-units\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625609    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-slash\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-cnibin\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625658    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-cni-bin\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-kubelet\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-etc-kubernetes\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-kubelet-config\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0b68f5-956e-4884-8f76-7ee1920966e1-tmp\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllkg\" (UniqueName: \"kubernetes.io/projected/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-kube-api-access-lllkg\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625827    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-system-cni-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-conf-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-run\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-tuned\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-kubelet\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-run-netns\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.625968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-node-log\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.626013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-cni-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.626061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-multus-certs\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.626091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.626929 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.626118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-socket-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.627706 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.626141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysctl-d\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.627706 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.626178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfrk\" (UniqueName: \"kubernetes.io/projected/a9557f36-6cf8-46fb-b997-9f9ade8915be-kube-api-access-hvfrk\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.677755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.677718    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:19.709962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.709917    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-whgvt"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-var-lib-kubelet\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-cni-bin\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-cni-bin\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-var-lib-kubelet\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-cni-multus\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e806d7b-ef85-4912-912f-428b60ebc744-multus-daemon-config\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-cni-multus\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.726761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-kubernetes\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpglz\" (UniqueName: \"kubernetes.io/projected/59ba2826-88d8-472b-8c1d-2f9d06841f6d-kube-api-access-gpglz\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" event={"ID":"4abd5f1ae2c9430a917972c6d312f06a","Type":"ContainerStarted","Data":"9628a037220a6a4bc8738b51d05dea8b951a6588ba423cdd91807a2e7942ec7d"}
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c1752e-456c-417e-9ef1-25e6efc078d1-iptables-alerter-script\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-socket-dir-parent\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-sys-fs\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.726988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4fb\" (UniqueName: \"kubernetes.io/projected/aa0b68f5-956e-4884-8f76-7ee1920966e1-kube-api-access-rv4fb\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-run-ovn-kubernetes\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c1752e-456c-417e-9ef1-25e6efc078d1-host-slash\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-netns\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngb75\" (UniqueName: \"kubernetes.io/projected/fbc60926-841c-4824-872b-30a7c806ade0-kube-api-access-ngb75\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysconfig\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e806d7b-ef85-4912-912f-428b60ebc744-cni-binary-copy\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-dbus\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-etc-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.727362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-cni-netd\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovnkube-config\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovn-node-metrics-cert\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74b6bb2c-1e44-46bd-90c8-e6c1772d35f9-konnectivity-ca\") pod \"konnectivity-agent-wb25w\" (UID: \"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9\") " pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e806d7b-ef85-4912-912f-428b60ebc744-multus-daemon-config\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c1752e-456c-417e-9ef1-25e6efc078d1-iptables-alerter-script\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727486    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-kubernetes\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-run-ovn-kubernetes\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-sys\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-sys\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-systemd\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-socket-dir-parent\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-dbus\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-systemd\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-sys-fs\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.728240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.727972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-systemd\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e806d7b-ef85-4912-912f-428b60ebc744-cni-binary-copy\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.728097    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c1752e-456c-417e-9ef1-25e6efc078d1-host-slash\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-netns\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-lib-modules\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-systemd-units\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.728247    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.228226424 +0000 UTC m=+4.676056861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-systemd-units\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-slash\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-cnibin\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-cni-bin\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728334    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-kubelet\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728373    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovnkube-config\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-etc-kubernetes\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-etc-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-lib-modules\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysconfig\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-kubelet-config\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-kubelet-config\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0b68f5-956e-4884-8f76-7ee1920966e1-tmp\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllkg\" (UniqueName: \"kubernetes.io/projected/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-kube-api-access-lllkg\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-system-cni-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-conf-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728607    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-etc-kubernetes\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-run\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-slash\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-kubelet\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-var-lib-cni-bin\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728689    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-cnibin\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-cni-netd\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-systemd\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-run\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.730110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-system-cni-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-tuned\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-kubelet\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.728995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-run-netns\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-node-log\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-cni-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-multus-certs\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-socket-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74b6bb2c-1e44-46bd-90c8-e6c1772d35f9-konnectivity-ca\") pod \"konnectivity-agent-wb25w\" (UID: \"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9\") " pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysctl-d\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfrk\" (UniqueName: \"kubernetes.io/projected/a9557f36-6cf8-46fb-b997-9f9ade8915be-kube-api-access-hvfrk\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-kubelet\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-registration-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.729315    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.731021 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.729363    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.229345746 +0000 UTC m=+4.677176196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-multus-certs\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-k8s-cni-cncf-io\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-run-netns\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-hostroot\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729485    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-node-log\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-cni-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-registration-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589b2\" (UniqueName: \"kubernetes.io/projected/3345f800-626d-4db5-b7b5-38d77107cc0b-kube-api-access-589b2\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysctl-d\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-host\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-var-lib-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-ovn\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-modprobe-d\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.731940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-socket-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9557f36-6cf8-46fb-b997-9f9ade8915be-host\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-multus-conf-dir\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p7h\" (UniqueName: \"kubernetes.io/projected/a5c1752e-456c-417e-9ef1-25e6efc078d1-kube-api-access-45p7h\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.729997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730070    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" event={"ID":"b4af3ae50a7b4cbff8bf701106a65509","Type":"ContainerStarted","Data":"2a2982e04b26ae8eac3fd65d44cab0ad53985a2495ba5b2ac92863e1f3542c6e"}
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mftd\" (UniqueName: \"kubernetes.io/projected/1e806d7b-ef85-4912-912f-428b60ebc744-kube-api-access-4mftd\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9557f36-6cf8-46fb-b997-9f9ade8915be-host\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-env-overrides\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cnibin\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9557f36-6cf8-46fb-b997-9f9ade8915be-serviceca\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74b6bb2c-1e44-46bd-90c8-e6c1772d35f9-agent-certs\") pod \"konnectivity-agent-wb25w\" (UID: \"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9\") " pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-var-lib-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.732817 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-device-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-log-socket\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-ovn\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-host-run-k8s-cni-cncf-io\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovnkube-script-lib\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysctl-conf\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-sysctl-conf\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9557f36-6cf8-46fb-b997-9f9ade8915be-serviceca\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730932    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-modprobe-d\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.730985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa0b68f5-956e-4884-8f76-7ee1920966e1-host\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-run-openvswitch\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-env-overrides\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-log-socket\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e806d7b-ef85-4912-912f-428b60ebc744-hostroot\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731198    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cnibin\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.733743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731303    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3345f800-626d-4db5-b7b5-38d77107cc0b-device-dir\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.734607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/59ba2826-88d8-472b-8c1d-2f9d06841f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.734607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731746    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-whgvt"
Sep 17 17:34:19.734607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.731920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovnkube-script-lib\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.734607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.732587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-ovn-node-metrics-cert\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.734607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.734175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0b68f5-956e-4884-8f76-7ee1920966e1-tmp\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.734607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.734514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aa0b68f5-956e-4884-8f76-7ee1920966e1-etc-tuned\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.736018 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.735961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74b6bb2c-1e44-46bd-90c8-e6c1772d35f9-agent-certs\") pod \"konnectivity-agent-wb25w\" (UID: \"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9\") " pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:19.752011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.751960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpglz\" (UniqueName: \"kubernetes.io/projected/59ba2826-88d8-472b-8c1d-2f9d06841f6d-kube-api-access-gpglz\") pod \"multus-additional-cni-plugins-wzs8n\" (UID: \"59ba2826-88d8-472b-8c1d-2f9d06841f6d\") " pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.775533 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.775492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4fb\" (UniqueName: \"kubernetes.io/projected/aa0b68f5-956e-4884-8f76-7ee1920966e1-kube-api-access-rv4fb\") pod \"tuned-qpq7l\" (UID: \"aa0b68f5-956e-4884-8f76-7ee1920966e1\") " pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.775533 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.775626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngb75\" (UniqueName: \"kubernetes.io/projected/fbc60926-841c-4824-872b-30a7c806ade0-kube-api-access-ngb75\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:19.775891 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.775737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p7h\" (UniqueName: \"kubernetes.io/projected/a5c1752e-456c-417e-9ef1-25e6efc078d1-kube-api-access-45p7h\") pod \"iptables-alerter-b9dts\" (UID: \"a5c1752e-456c-417e-9ef1-25e6efc078d1\") " pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.779889 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.779861    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:19.780074 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.779895    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:19.780074 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.779910    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jjtts for pod openshift-network-diagnostics/network-check-target-7c8nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:19.780074 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:19.779978    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts podName:0fa66574-a870-4c4a-9932-e850183522b4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.279955566 +0000 UTC m=+4.727786018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jjtts" (UniqueName: "kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts") pod "network-check-target-7c8nf" (UID: "0fa66574-a870-4c4a-9932-e850183522b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:19.782000 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.781975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfrk\" (UniqueName: \"kubernetes.io/projected/a9557f36-6cf8-46fb-b997-9f9ade8915be-kube-api-access-hvfrk\") pod \"node-ca-j9mf9\" (UID: \"a9557f36-6cf8-46fb-b997-9f9ade8915be\") " pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.782309 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.782281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mftd\" (UniqueName: \"kubernetes.io/projected/1e806d7b-ef85-4912-912f-428b60ebc744-kube-api-access-4mftd\") pod \"multus-fz8ls\" (UID: \"1e806d7b-ef85-4912-912f-428b60ebc744\") " pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.782763 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.782739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-589b2\" (UniqueName: \"kubernetes.io/projected/3345f800-626d-4db5-b7b5-38d77107cc0b-kube-api-access-589b2\") pod \"aws-ebs-csi-driver-node-zj5lq\" (UID: \"3345f800-626d-4db5-b7b5-38d77107cc0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.783182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.783156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllkg\" (UniqueName: \"kubernetes.io/projected/7cde33cc-1dab-4ddc-9c17-1ff58962ee01-kube-api-access-lllkg\") pod \"ovnkube-node-5l8m4\" (UID: \"7cde33cc-1dab-4ddc-9c17-1ff58962ee01\") " pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.889946 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.889916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzs8n"
Sep 17 17:34:19.899367 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.899304    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fz8ls"
Sep 17 17:34:19.910024 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.909986    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq"
Sep 17 17:34:19.917160 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.917119    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qpq7l"
Sep 17 17:34:19.927145 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.927108    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j9mf9"
Sep 17 17:34:19.936675 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.936649    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b9dts"
Sep 17 17:34:19.952227 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.952200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:19.959086 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:19.959063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:20.234542 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.234436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:20.234542 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.234506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:20.234770 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.234612    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:20.234770 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.234637    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:20.234770 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.234700    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:21.234678836 +0000 UTC m=+5.682509291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:20.234770 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.234721    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:21.234711085 +0000 UTC m=+5.682541522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:20.335600 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.335557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:20.335800 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.335742    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:20.335800 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.335769    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:20.335800 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.335787    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jjtts for pod openshift-network-diagnostics/network-check-target-7c8nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:20.335935 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.335862    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts podName:0fa66574-a870-4c4a-9932-e850183522b4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:21.335841935 +0000 UTC m=+5.783672391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjtts" (UniqueName: "kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts") pod "network-check-target-7c8nf" (UID: "0fa66574-a870-4c4a-9932-e850183522b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:20.721857 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.721754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:20.721857 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.721795    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:20.722617 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.721906    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:20.722617 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:20.722015    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:20.736255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.736222    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:19 +0000 UTC" deadline="2028-03-17 20:54:25.762866135 +0000 UTC"
Sep 17 17:34:20.736255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:20.736251    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13131h20m5.026619118s"
Sep 17 17:34:20.959847 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:20.959811    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e806d7b_ef85_4912_912f_428b60ebc744.slice/crio-f2e25e182a75520507f582cbcdd4efcbe8285667f05afe15b04b8dec3954cf6d WatchSource:0}: Error finding container f2e25e182a75520507f582cbcdd4efcbe8285667f05afe15b04b8dec3954cf6d: Status 404 returned error can't find the container with id f2e25e182a75520507f582cbcdd4efcbe8285667f05afe15b04b8dec3954cf6d
Sep 17 17:34:20.961585 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:20.961374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9557f36_6cf8_46fb_b997_9f9ade8915be.slice/crio-0d1b1dbdbfd963e467cdac496bbd41e274692e22d02ed05e2b7054c5e6f6fcbe WatchSource:0}: Error finding container 0d1b1dbdbfd963e467cdac496bbd41e274692e22d02ed05e2b7054c5e6f6fcbe: Status 404 returned error can't find the container with id 0d1b1dbdbfd963e467cdac496bbd41e274692e22d02ed05e2b7054c5e6f6fcbe
Sep 17 17:34:20.961853 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:20.961832    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0b68f5_956e_4884_8f76_7ee1920966e1.slice/crio-e4b682bf63f6154d09e375d93202ef060b8e40ac4d7d6d033da7c0a480219e2e WatchSource:0}: Error finding container e4b682bf63f6154d09e375d93202ef060b8e40ac4d7d6d033da7c0a480219e2e: Status 404 returned error can't find the container with id e4b682bf63f6154d09e375d93202ef060b8e40ac4d7d6d033da7c0a480219e2e
Sep 17 17:34:20.964790 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:20.964768    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cde33cc_1dab_4ddc_9c17_1ff58962ee01.slice/crio-57ded9fe625ad5c328c4ec8f66195390f01a23a0acd2097c746c7e06580ecf8c WatchSource:0}: Error finding container 57ded9fe625ad5c328c4ec8f66195390f01a23a0acd2097c746c7e06580ecf8c: Status 404 returned error can't find the container with id 57ded9fe625ad5c328c4ec8f66195390f01a23a0acd2097c746c7e06580ecf8c
Sep 17 17:34:20.967543 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:20.967505    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c1752e_456c_417e_9ef1_25e6efc078d1.slice/crio-e666471f1c26682f012ddee49a100c236a783bd9824304a8ce00da9f881c9471 WatchSource:0}: Error finding container e666471f1c26682f012ddee49a100c236a783bd9824304a8ce00da9f881c9471: Status 404 returned error can't find the container with id e666471f1c26682f012ddee49a100c236a783bd9824304a8ce00da9f881c9471
Sep 17 17:34:20.969150 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:20.968530    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ba2826_88d8_472b_8c1d_2f9d06841f6d.slice/crio-1ce3ecf08cf9a305c59e56f92e69badc5f8e974ac0207a858c23abeb00bac777 WatchSource:0}: Error finding container 1ce3ecf08cf9a305c59e56f92e69badc5f8e974ac0207a858c23abeb00bac777: Status 404 returned error can't find the container with id 1ce3ecf08cf9a305c59e56f92e69badc5f8e974ac0207a858c23abeb00bac777
Sep 17 17:34:21.242718 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.242622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:21.242718 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.242679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:21.242953 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.242788    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:21.242953 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.242815    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:21.242953 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.242878    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:23.242846176 +0000 UTC m=+7.690676617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:21.242953 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.242894    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:23.242887541 +0000 UTC m=+7.690717980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:21.343644 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.343603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:21.343824 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.343809    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:21.343883 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.343836    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:21.343883 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.343852    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jjtts for pod openshift-network-diagnostics/network-check-target-7c8nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:21.344003 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.343941    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts podName:0fa66574-a870-4c4a-9932-e850183522b4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:23.343921344 +0000 UTC m=+7.791751784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjtts" (UniqueName: "kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts") pod "network-check-target-7c8nf" (UID: "0fa66574-a870-4c4a-9932-e850183522b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:21.725208 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.724295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:21.725208 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:21.724728    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:21.736720 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.736681    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:19 +0000 UTC" deadline="2028-03-25 11:44:45.729201897 +0000 UTC"
Sep 17 17:34:21.736720 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.736719    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13314h10m23.992487858s"
Sep 17 17:34:21.746370 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.740657    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4af3ae50a7b4cbff8bf701106a65509" containerID="7674a691513528bf8696ad915797c5e247c0acbffae1d313fcacb54ae6f4f0b1" exitCode=0
Sep 17 17:34:21.748350 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.748114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" event={"ID":"b4af3ae50a7b4cbff8bf701106a65509","Type":"ContainerDied","Data":"7674a691513528bf8696ad915797c5e247c0acbffae1d313fcacb54ae6f4f0b1"}
Sep 17 17:34:21.775691 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.775657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wb25w" event={"ID":"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9","Type":"ContainerStarted","Data":"e8c5ea6256e2c79556d5936c1b1b1f12a47c64d58caa8ce528f51a6ece58ec75"}
Sep 17 17:34:21.791305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.791250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq" event={"ID":"3345f800-626d-4db5-b7b5-38d77107cc0b","Type":"ContainerStarted","Data":"12bd969399f234d5546282830df9ecac07582b2242a047b014e8d266fc89ea7a"}
Sep 17 17:34:21.798151 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.798110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerStarted","Data":"1ce3ecf08cf9a305c59e56f92e69badc5f8e974ac0207a858c23abeb00bac777"}
Sep 17 17:34:21.803924 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.803888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qpq7l" event={"ID":"aa0b68f5-956e-4884-8f76-7ee1920966e1","Type":"ContainerStarted","Data":"e4b682bf63f6154d09e375d93202ef060b8e40ac4d7d6d033da7c0a480219e2e"}
Sep 17 17:34:21.809158 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.809114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" event={"ID":"4abd5f1ae2c9430a917972c6d312f06a","Type":"ContainerStarted","Data":"2a156a6c148472de1fedfa16802c741ed01c3130bb8206b7e1018eb23c8f33ee"}
Sep 17 17:34:21.813769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.813734    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b9dts" event={"ID":"a5c1752e-456c-417e-9ef1-25e6efc078d1","Type":"ContainerStarted","Data":"e666471f1c26682f012ddee49a100c236a783bd9824304a8ce00da9f881c9471"}
Sep 17 17:34:21.816537 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.816510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"57ded9fe625ad5c328c4ec8f66195390f01a23a0acd2097c746c7e06580ecf8c"}
Sep 17 17:34:21.829740 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.829705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j9mf9" event={"ID":"a9557f36-6cf8-46fb-b997-9f9ade8915be","Type":"ContainerStarted","Data":"0d1b1dbdbfd963e467cdac496bbd41e274692e22d02ed05e2b7054c5e6f6fcbe"}
Sep 17 17:34:21.848533 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:21.848496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fz8ls" event={"ID":"1e806d7b-ef85-4912-912f-428b60ebc744","Type":"ContainerStarted","Data":"f2e25e182a75520507f582cbcdd4efcbe8285667f05afe15b04b8dec3954cf6d"}
Sep 17 17:34:22.722283 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:22.722241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:22.722495 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:22.722394    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:22.722814 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:22.722790    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:22.722911 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:22.722891    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:22.857445 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:22.857406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" event={"ID":"b4af3ae50a7b4cbff8bf701106a65509","Type":"ContainerStarted","Data":"e253e52c9a27f8dd9ec5ee0bff6a81ac956324a4b78105b2f86b1df007a40780"}
Sep 17 17:34:22.877004 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:22.876885    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-71.ec2.internal" podStartSLOduration=3.876858382 podStartE2EDuration="3.876858382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:21.828739208 +0000 UTC m=+6.276569683" watchObservedRunningTime="2026-09-17 17:34:22.876858382 +0000 UTC m=+7.324688839"
Sep 17 17:34:23.260575 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:23.260533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:23.260775 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:23.260637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:23.260775 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.260763    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:23.260897 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.260831    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.260812709 +0000 UTC m=+11.708643145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:23.261412 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.261389    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:23.261509 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.261448    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.261432682 +0000 UTC m=+11.709263120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:23.361132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:23.361091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:23.361422 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.361318    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:23.361422 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.361357    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:23.361422 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.361370    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jjtts for pod openshift-network-diagnostics/network-check-target-7c8nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:23.361594 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.361495    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts podName:0fa66574-a870-4c4a-9932-e850183522b4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.361415902 +0000 UTC m=+11.809246342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjtts" (UniqueName: "kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts") pod "network-check-target-7c8nf" (UID: "0fa66574-a870-4c4a-9932-e850183522b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:23.722236 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:23.721651    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:23.722236 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:23.721807    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:24.722223 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:24.721535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:24.722223 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:24.721670    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:24.722223 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:24.722087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:24.722223 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:24.722175    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:25.725740 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:25.725710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:25.726186 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:25.725861    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:26.722620 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:26.722446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:26.722620 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:26.722483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:26.722620 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:26.722588    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:26.723077 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:26.723050    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:27.297366 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:27.297315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:27.297861 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:27.297390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:27.297861 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.297530    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:27.297861 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.297595    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:35.297575274 +0000 UTC m=+19.745405708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:27.297861 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.297697    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:27.297861 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.297746    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:35.297733244 +0000 UTC m=+19.745563692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:27.398093 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:27.398001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:27.398314 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.398203    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:27.398314 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.398233    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:27.398314 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.398251    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jjtts for pod openshift-network-diagnostics/network-check-target-7c8nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:27.398583 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.398319    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts podName:0fa66574-a870-4c4a-9932-e850183522b4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:35.398299538 +0000 UTC m=+19.846129973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjtts" (UniqueName: "kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts") pod "network-check-target-7c8nf" (UID: "0fa66574-a870-4c4a-9932-e850183522b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:27.722470 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:27.722373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:27.722628 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:27.722498    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:28.722343 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:28.722273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:28.722343 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:28.722302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:28.722815 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:28.722428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:28.722961 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:28.722902    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:29.721399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:29.721367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:29.721587 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:29.721494    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:30.722112 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:30.722067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:30.722112 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:30.722094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:30.722603 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:30.722206    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:30.722603 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:30.722311    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:31.721548 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:31.721515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:31.721722 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:31.721648    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:32.721581 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:32.721546    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:32.722011 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:32.721682    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:32.722011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:32.721720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:32.722011 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:32.721842    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:33.722039 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:33.722005    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:33.722438 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:33.722143    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:34.722280 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:34.722245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:34.722660 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:34.722377    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:34.722946 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:34.722686    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:34.723079 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:34.723056    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:35.352853 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:35.352819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:35.353042 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:35.352874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:35.353042 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.352988    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:35.353042 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.353005    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:35.353167 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.353058    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.3530396 +0000 UTC m=+35.800870035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:35.353167 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.353078    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.353067703 +0000 UTC m=+35.800898140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:35.453214 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:35.453180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:35.453394 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.453372    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:35.453448 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.453400    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:35.453448 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.453411    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jjtts for pod openshift-network-diagnostics/network-check-target-7c8nf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:35.453524 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.453462    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts podName:0fa66574-a870-4c4a-9932-e850183522b4 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.453448476 +0000 UTC m=+35.901278909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjtts" (UniqueName: "kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts") pod "network-check-target-7c8nf" (UID: "0fa66574-a870-4c4a-9932-e850183522b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:35.722440 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:35.722364    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:35.722790 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:35.722512    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:36.722212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:36.722174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:36.722412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:36.722174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:36.722412 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:36.722300    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:36.722412 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:36.722377    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:37.722158 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:37.722122    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:37.722358 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:37.722236    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:38.721701 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.721490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:38.722365 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.721549    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:38.722365 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:38.721729    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:38.722365 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:38.721804    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:38.890911 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.890868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq" event={"ID":"3345f800-626d-4db5-b7b5-38d77107cc0b","Type":"ContainerStarted","Data":"8687a1eb59ab5fc78b8a86b9e4c71027be1c6f3a5abaa8b27283690b2c71053d"}
Sep 17 17:34:38.892613 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.892583    2562 generic.go:356] "Generic (PLEG): container finished" podID="59ba2826-88d8-472b-8c1d-2f9d06841f6d" containerID="44b1a23e5fe286f9651141ef8abc879732d66eb8f0fb7313243b724fdd409c3b" exitCode=0
Sep 17 17:34:38.892732 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.892673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerDied","Data":"44b1a23e5fe286f9651141ef8abc879732d66eb8f0fb7313243b724fdd409c3b"}
Sep 17 17:34:38.894699 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.894677    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qpq7l" event={"ID":"aa0b68f5-956e-4884-8f76-7ee1920966e1","Type":"ContainerStarted","Data":"364136dd9a71bfbf259681a20c2d2657f0fa32d216ab3b38fec9722743c8efbd"}
Sep 17 17:34:38.897844 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.897814    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"d223555a16837c9d812c5f834b73835d12dd8ecfcfff2640d081e5f4e9b313d1"}
Sep 17 17:34:38.897991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.897888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"318379b00077d6f4dbf4eb544781ecc991a9a1afe33d037f9657abaa8bd75c49"}
Sep 17 17:34:38.897991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.897939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"8866817fc2f7975b1b1efffcbba7c7be1711bdbcc8c2512c90700a013555dc4b"}
Sep 17 17:34:38.897991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.897968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"a5ff2cebabc79f1574af7dfa4ad30c0c85b6640738bffd487f74e91b94469afc"}
Sep 17 17:34:38.898131 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.898017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"7ed2877aa5e1266eed6d9b325715d2f20729dcb786b4c44fab22dfddeb2a8b2f"}
Sep 17 17:34:38.898131 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.898066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"2b30b3670cd10420e2d94309603240f25adde55885b2b95befc9fea572596662"}
Sep 17 17:34:38.899183 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.899150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j9mf9" event={"ID":"a9557f36-6cf8-46fb-b997-9f9ade8915be","Type":"ContainerStarted","Data":"cba7df85c4a7b7d9dbe2e37975b5822dc816a07654ddce091aa00c3eb441a9d9"}
Sep 17 17:34:38.900653 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.900630    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fz8ls" event={"ID":"1e806d7b-ef85-4912-912f-428b60ebc744","Type":"ContainerStarted","Data":"8e2b8a9dbb49df7e51c9c494d48b877fe5d261f7194df96bb54ebfd572ca584c"}
Sep 17 17:34:38.901856 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.901837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wb25w" event={"ID":"74b6bb2c-1e44-46bd-90c8-e6c1772d35f9","Type":"ContainerStarted","Data":"31d50f5f15187c5839ae91f1db79edb4e827469ebf04b203f9d3d864a042850b"}
Sep 17 17:34:38.916581 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.916536    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-71.ec2.internal" podStartSLOduration=19.916523456 podStartE2EDuration="19.916523456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:22.877548864 +0000 UTC m=+7.325379321" watchObservedRunningTime="2026-09-17 17:34:38.916523456 +0000 UTC m=+23.364353911"
Sep 17 17:34:38.951675 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.951617    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qpq7l" podStartSLOduration=3.774743561 podStartE2EDuration="20.95159326s" totalImagesPullingTime="17.176849699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:38.937017309 +0000 UTC m=+23.384847785" watchObservedRunningTime="2026-09-17 17:34:38.95159326 +0000 UTC m=+23.399423716"
Sep 17 17:34:38.951843 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.951760    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j9mf9" podStartSLOduration=7.701910135 podStartE2EDuration="20.951749655s" totalImagesPullingTime="13.24983952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:38.951379657 +0000 UTC m=+23.399210133" watchObservedRunningTime="2026-09-17 17:34:38.951749655 +0000 UTC m=+23.399580111"
Sep 17 17:34:38.998392 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:38.998279    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fz8ls" podStartSLOduration=3.787713944 podStartE2EDuration="20.99826264s" totalImagesPullingTime="17.210548696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:38.97523368 +0000 UTC m=+23.423064165" watchObservedRunningTime="2026-09-17 17:34:38.99826264 +0000 UTC m=+23.446093094"
Sep 17 17:34:39.208761 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.208741    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:39.681165 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.680956    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:39.208757299Z","UUID":"9a11da83-aa77-4caf-870d-02a5f79180d2","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:39.683041 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.683017    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:39.683180 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.683062    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:39.723834 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.723802    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:39.724274 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:39.723942    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:39.907208 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.907127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq" event={"ID":"3345f800-626d-4db5-b7b5-38d77107cc0b","Type":"ContainerStarted","Data":"68a64fa85fe527260e85ba18fdbc8cf6395cd71e5714b116e9f1b5d25b10fdb4"}
Sep 17 17:34:39.909013 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.908957    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b9dts" event={"ID":"a5c1752e-456c-417e-9ef1-25e6efc078d1","Type":"ContainerStarted","Data":"f32d8a9f35a90f4d5b66ced6b19af34fde21006c922289b1aa0522abdf469e96"}
Sep 17 17:34:39.927086 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:39.926892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wb25w" podStartSLOduration=5.17266914 podStartE2EDuration="21.926871485s" totalImagesPullingTime="16.754202345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:38.998520014 +0000 UTC m=+23.446350506" watchObservedRunningTime="2026-09-17 17:34:39.926871485 +0000 UTC m=+24.374701945"
Sep 17 17:34:40.722046 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:40.722014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:40.722193 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:40.722171    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:40.722255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:40.722207    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:40.722364 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:40.722318    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:40.913680 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:40.913595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq" event={"ID":"3345f800-626d-4db5-b7b5-38d77107cc0b","Type":"ContainerStarted","Data":"f420061524ce7df263c1d27e5fa3e662748999df8bb868e3b54d9959f409482c"}
Sep 17 17:34:40.917356 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:40.917313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"db418261a76d2eee61b00a0745a8f3871c6a7ca742b3372117ac806add0b8eee"}
Sep 17 17:34:40.975603 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:40.975545    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b9dts" podStartSLOduration=5.803918201 podStartE2EDuration="22.975529438s" totalImagesPullingTime="17.171611237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:39.92744925 +0000 UTC m=+24.375279737" watchObservedRunningTime="2026-09-17 17:34:40.975529438 +0000 UTC m=+25.423359909"
Sep 17 17:34:41.633786 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:41.633748    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:41.634505 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:41.634483    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:41.658913 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:41.658864    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zj5lq" podStartSLOduration=4.913567477 podStartE2EDuration="23.658846876s" totalImagesPullingTime="18.745279399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:40.975772644 +0000 UTC m=+25.423603126" watchObservedRunningTime="2026-09-17 17:34:41.658846876 +0000 UTC m=+26.106677331"
Sep 17 17:34:41.722206 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:41.722172    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:41.722376 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:41.722291    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:42.721982 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.721947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:42.721982 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.721979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:42.722558 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:42.722070    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:42.722558 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:42.722173    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:42.925755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.925726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" event={"ID":"7cde33cc-1dab-4ddc-9c17-1ff58962ee01","Type":"ContainerStarted","Data":"07c6cd4ae191b56f980c2dc9f82fe885b6eaaedca8b1a231a5201d7fc3ebbac8"}
Sep 17 17:34:42.925986 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.925948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:42.926091 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.926004    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:42.940779 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.940754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:42.940895 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.940851    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:42.970514 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:42.970356    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4" podStartSLOduration=7.731057654 podStartE2EDuration="24.970341151s" totalImagesPullingTime="17.239283497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:42.969254499 +0000 UTC m=+27.417084955" watchObservedRunningTime="2026-09-17 17:34:42.970341151 +0000 UTC m=+27.418171600"
Sep 17 17:34:43.722124 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:43.722089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:43.722673 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:43.722206    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:43.929740 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:43.929705    2562 generic.go:356] "Generic (PLEG): container finished" podID="59ba2826-88d8-472b-8c1d-2f9d06841f6d" containerID="91345b03e15ec6ab19cff96df48bc15e946b383d4de092ab85d9807ce26510bd" exitCode=0
Sep 17 17:34:43.929918 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:43.929793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerDied","Data":"91345b03e15ec6ab19cff96df48bc15e946b383d4de092ab85d9807ce26510bd"}
Sep 17 17:34:43.929918 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:43.929860    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:44.607392 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:44.607364    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:34:44.722102 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:44.722071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:44.722252 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:44.722071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:44.722252 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:44.722196    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:44.722759 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:44.722264    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:44.934670 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:44.934639    2562 generic.go:356] "Generic (PLEG): container finished" podID="59ba2826-88d8-472b-8c1d-2f9d06841f6d" containerID="e64aecf57cae9201507a44733b138a6e167e4982c8564c6b6feb09edc84406d0" exitCode=0
Sep 17 17:34:44.934754 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:44.934717    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerDied","Data":"e64aecf57cae9201507a44733b138a6e167e4982c8564c6b6feb09edc84406d0"}
Sep 17 17:34:45.064530 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:45.064503    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rzpqd"]
Sep 17 17:34:45.064662 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:45.064593    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:45.064697 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:45.064676    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:45.068040 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:45.068014    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2v8l"]
Sep 17 17:34:45.068167 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:45.068096    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:45.068266 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:45.068243    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:45.071904 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:45.071852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7c8nf"]
Sep 17 17:34:45.072045 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:45.071982    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:45.074579 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:45.072168    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:46.722093 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.722062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:46.722593 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.722070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:46.722593 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:46.722170    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:46.722593 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.722190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:46.722593 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:46.722255    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:46.722593 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:46.722309    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:46.803447 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.803416    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-mwgsw"]
Sep 17 17:34:46.829176 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.829152    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:46.832610 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.832590    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:46.833119 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.833100    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:46.833203 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.833191    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qqsk8\""
Sep 17 17:34:46.932476 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.932442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ebdeed2-06ee-41de-b637-0ae050cf8c63-hosts-file\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:46.932622 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.932502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs68f\" (UniqueName: \"kubernetes.io/projected/6ebdeed2-06ee-41de-b637-0ae050cf8c63-kube-api-access-fs68f\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:46.932622 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.932536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebdeed2-06ee-41de-b637-0ae050cf8c63-tmp-dir\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:46.941151 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.941126    2562 generic.go:356] "Generic (PLEG): container finished" podID="59ba2826-88d8-472b-8c1d-2f9d06841f6d" containerID="6ee11ec95396846b0a9f4bcc87bc1c2dbece323fdb42a4bda6b8b5319d95cd0f" exitCode=0
Sep 17 17:34:46.941325 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:46.941175    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerDied","Data":"6ee11ec95396846b0a9f4bcc87bc1c2dbece323fdb42a4bda6b8b5319d95cd0f"}
Sep 17 17:34:47.033621 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.033533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs68f\" (UniqueName: \"kubernetes.io/projected/6ebdeed2-06ee-41de-b637-0ae050cf8c63-kube-api-access-fs68f\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.033621 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.033607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebdeed2-06ee-41de-b637-0ae050cf8c63-tmp-dir\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.033809 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.033659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ebdeed2-06ee-41de-b637-0ae050cf8c63-hosts-file\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.033809 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.033766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ebdeed2-06ee-41de-b637-0ae050cf8c63-hosts-file\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.034083 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.034066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebdeed2-06ee-41de-b637-0ae050cf8c63-tmp-dir\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.048440 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.048418    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs68f\" (UniqueName: \"kubernetes.io/projected/6ebdeed2-06ee-41de-b637-0ae050cf8c63-kube-api-access-fs68f\") pod \"node-resolver-mwgsw\" (UID: \"6ebdeed2-06ee-41de-b637-0ae050cf8c63\") " pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.139255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.139226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mwgsw"
Sep 17 17:34:47.152829 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:47.152794    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebdeed2_06ee_41de_b637_0ae050cf8c63.slice/crio-f8cb2ed0bd864972fd49e3c3df08b9a0094c04ee0348dccab72792c85a4e46b0 WatchSource:0}: Error finding container f8cb2ed0bd864972fd49e3c3df08b9a0094c04ee0348dccab72792c85a4e46b0: Status 404 returned error can't find the container with id f8cb2ed0bd864972fd49e3c3df08b9a0094c04ee0348dccab72792c85a4e46b0
Sep 17 17:34:47.945539 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.945497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mwgsw" event={"ID":"6ebdeed2-06ee-41de-b637-0ae050cf8c63","Type":"ContainerStarted","Data":"e05f0632fd879c73b86dd9b6c1ed5af166f3e2b796780abaa76f5f0fed3bcf25"}
Sep 17 17:34:47.946092 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.945567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mwgsw" event={"ID":"6ebdeed2-06ee-41de-b637-0ae050cf8c63","Type":"ContainerStarted","Data":"f8cb2ed0bd864972fd49e3c3df08b9a0094c04ee0348dccab72792c85a4e46b0"}
Sep 17 17:34:47.970624 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:47.970571    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mwgsw" podStartSLOduration=1.970554731 podStartE2EDuration="1.970554731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:47.970476218 +0000 UTC m=+32.418306698" watchObservedRunningTime="2026-09-17 17:34:47.970554731 +0000 UTC m=+32.418385186"
Sep 17 17:34:48.721568 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:48.721521    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:48.721568 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:48.721546    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:48.721804 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:48.721550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:48.721804 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:48.721648    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:48.721804 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:48.721730    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:48.721948 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:48.721832    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:49.091039 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:49.090998    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:49.091467 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:49.091148    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:49.091665 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:49.091646    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wb25w"
Sep 17 17:34:50.722063 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:50.721834    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:50.722551 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:50.721899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:50.722551 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:50.722148    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d2v8l" podUID="6c83b0a2-9af0-4601-9d20-79b3a6533dbf"
Sep 17 17:34:50.722551 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:50.722258    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7c8nf" podUID="0fa66574-a870-4c4a-9932-e850183522b4"
Sep 17 17:34:50.722551 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:50.721910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:50.722551 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:50.722383    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:34:51.003612 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.003582    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-71.ec2.internal" event="NodeReady"
Sep 17 17:34:51.003764 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.003752    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:51.048097 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.048028    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"]
Sep 17 17:34:51.062096 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.062071    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"]
Sep 17 17:34:51.062225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.062204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.066623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.066593    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:34:51.066623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.066615    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:34:51.067026 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.067007    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:34:51.067128 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.067008    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6xvsk\""
Sep 17 17:34:51.075550 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.075526    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:34:51.076239 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.076211    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"]
Sep 17 17:34:51.076378 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.076360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.080514 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.080488    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:34:51.080625 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.080601    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:34:51.080766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.080725    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-95zq5\""
Sep 17 17:34:51.088891 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.088861    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"]
Sep 17 17:34:51.089033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.088899    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7swlh"]
Sep 17 17:34:51.089033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.088939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"
Sep 17 17:34:51.091749 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.091731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:51.092034 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.092015    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-74wbs\""
Sep 17 17:34:51.092113 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.092017    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:51.101003 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.100983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"]
Sep 17 17:34:51.101101 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.101011    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bljxr"]
Sep 17 17:34:51.101101 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.101087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.104804 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.104786    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:51.104995 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.104980    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:51.105133 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.105118    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjhsr\""
Sep 17 17:34:51.105434 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.105416    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:51.113343 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.113306    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"]
Sep 17 17:34:51.113426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.113352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7swlh"]
Sep 17 17:34:51.113426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.113374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bljxr"]
Sep 17 17:34:51.113426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.113402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.116846 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.116818    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:51.116935 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.116922    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqqq4\""
Sep 17 17:34:51.117255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.117241    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:51.163787 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.163757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82a1fb15-93eb-46b6-a334-5ca33f438d14-ca-trust-extracted\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.163916 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.163808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-image-registry-private-configuration\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.163916 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.163840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-trusted-ca\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.163916 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.163885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-installation-pull-secrets\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.164033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.163932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs26z\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-kube-api-access-zs26z\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.164033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.163994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04876813-f72f-4194-8dab-62304008b825-cert\") pod \"ingress-canary-7swlh\" (UID: \"04876813-f72f-4194-8dab-62304008b825\") " pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.164033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-certificates\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.164144 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xxz\" (UniqueName: \"kubernetes.io/projected/04876813-f72f-4194-8dab-62304008b825-kube-api-access-w5xxz\") pod \"ingress-canary-7swlh\" (UID: \"04876813-f72f-4194-8dab-62304008b825\") " pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.164144 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.164144 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-bound-sa-token\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.164144 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.164285 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.164285 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.164183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c78v\" (UniqueName: \"kubernetes.io/projected/2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e-kube-api-access-8c78v\") pod \"network-check-source-7cbc45fddc-dp2xf\" (UID: \"2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"
Sep 17 17:34:51.264645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.264620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.264744 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.264654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.264744 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.264671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c78v\" (UniqueName: \"kubernetes.io/projected/2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e-kube-api-access-8c78v\") pod \"network-check-source-7cbc45fddc-dp2xf\" (UID: \"2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"
Sep 17 17:34:51.264837 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.264814    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:51.264891 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.264869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c86541f5-fa09-4a99-b838-01322d9c0cb6-config-volume\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.264891 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.264886    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.764863791 +0000 UTC m=+36.212694247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:51.264987 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.264919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.264987 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.264970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86541f5-fa09-4a99-b838-01322d9c0cb6-tmp-dir\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.265064 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82a1fb15-93eb-46b6-a334-5ca33f438d14-ca-trust-extracted\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265109 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-image-registry-private-configuration\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265143 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-trusted-ca\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-installation-pull-secrets\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs26z\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-kube-api-access-zs26z\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04876813-f72f-4194-8dab-62304008b825-cert\") pod \"ingress-canary-7swlh\" (UID: \"04876813-f72f-4194-8dab-62304008b825\") " pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscp5\" (UniqueName: \"kubernetes.io/projected/c86541f5-fa09-4a99-b838-01322d9c0cb6-kube-api-access-bscp5\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-certificates\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xxz\" (UniqueName: \"kubernetes.io/projected/04876813-f72f-4194-8dab-62304008b825-kube-api-access-w5xxz\") pod \"ingress-canary-7swlh\" (UID: \"04876813-f72f-4194-8dab-62304008b825\") " pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-bound-sa-token\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.265558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.265535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82a1fb15-93eb-46b6-a334-5ca33f438d14-ca-trust-extracted\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.266167 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.265625    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:51.266167 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.265637    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:34:51.266167 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.265713    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.765695397 +0000 UTC m=+36.213525853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:34:51.266387 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.266244    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-certificates\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.266573 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.266551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-trusted-ca\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.269700 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.269676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-image-registry-private-configuration\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.269783 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.269767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04876813-f72f-4194-8dab-62304008b825-cert\") pod \"ingress-canary-7swlh\" (UID: \"04876813-f72f-4194-8dab-62304008b825\") " pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.270181 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.270161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-installation-pull-secrets\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.287106 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.287081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs26z\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-kube-api-access-zs26z\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.287842 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.287822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c78v\" (UniqueName: \"kubernetes.io/projected/2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e-kube-api-access-8c78v\") pod \"network-check-source-7cbc45fddc-dp2xf\" (UID: \"2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"
Sep 17 17:34:51.288357 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.288319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xxz\" (UniqueName: \"kubernetes.io/projected/04876813-f72f-4194-8dab-62304008b825-kube-api-access-w5xxz\") pod \"ingress-canary-7swlh\" (UID: \"04876813-f72f-4194-8dab-62304008b825\") " pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.288566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.288546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-bound-sa-token\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.366664 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.366640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c86541f5-fa09-4a99-b838-01322d9c0cb6-config-volume\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.366776 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.366677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.366776 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.366703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86541f5-fa09-4a99-b838-01322d9c0cb6-tmp-dir\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.366877 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.366809    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:51.366932 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.366876    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.866856967 +0000 UTC m=+36.314687414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:51.366932 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.366913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:51.367035 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.366950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscp5\" (UniqueName: \"kubernetes.io/projected/c86541f5-fa09-4a99-b838-01322d9c0cb6-kube-api-access-bscp5\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.367089 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.367074    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:51.367135 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.367088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c86541f5-fa09-4a99-b838-01322d9c0cb6-tmp-dir\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.367135 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.367093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:51.367215 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.367148    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:23.367130477 +0000 UTC m=+67.814961119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:51.367215 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.367154    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:51.367215 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.367197    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret podName:6c83b0a2-9af0-4601-9d20-79b3a6533dbf nodeName:}" failed. No retries permitted until 2026-09-17 17:35:23.367181314 +0000 UTC m=+67.815011749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret") pod "global-pull-secret-syncer-d2v8l" (UID: "6c83b0a2-9af0-4601-9d20-79b3a6533dbf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:51.367315 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.367270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c86541f5-fa09-4a99-b838-01322d9c0cb6-config-volume\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.389772 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.389747    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscp5\" (UniqueName: \"kubernetes.io/projected/c86541f5-fa09-4a99-b838-01322d9c0cb6-kube-api-access-bscp5\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.397656 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.397635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"
Sep 17 17:34:51.408627 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.408605    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7swlh"
Sep 17 17:34:51.467508 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.467475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:51.470077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.470053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtts\" (UniqueName: \"kubernetes.io/projected/0fa66574-a870-4c4a-9932-e850183522b4-kube-api-access-jjtts\") pod \"network-check-target-7c8nf\" (UID: \"0fa66574-a870-4c4a-9932-e850183522b4\") " pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:51.733554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.733525    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7swlh"]
Sep 17 17:34:51.735312 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.735286    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf"]
Sep 17 17:34:51.738352 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:51.738175    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e211d54_8b6d_4ae0_b999_ab1a3b22ed1e.slice/crio-859834e0e0ec51d736e8f1093b7e9a0cc2205f65fecc8cb7c23ff02fd3b130f6 WatchSource:0}: Error finding container 859834e0e0ec51d736e8f1093b7e9a0cc2205f65fecc8cb7c23ff02fd3b130f6: Status 404 returned error can't find the container with id 859834e0e0ec51d736e8f1093b7e9a0cc2205f65fecc8cb7c23ff02fd3b130f6
Sep 17 17:34:51.738650 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:51.738618    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04876813_f72f_4194_8dab_62304008b825.slice/crio-cb77e338306bccd7cd4a83c298b6ccbe1f14441c4cc2493afa289b590530a9c7 WatchSource:0}: Error finding container cb77e338306bccd7cd4a83c298b6ccbe1f14441c4cc2493afa289b590530a9c7: Status 404 returned error can't find the container with id cb77e338306bccd7cd4a83c298b6ccbe1f14441c4cc2493afa289b590530a9c7
Sep 17 17:34:51.768890 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.768868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:51.768959 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.768904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:51.769024 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.769013    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:51.769077 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.769068    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.769055402 +0000 UTC m=+37.216885835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:51.769118 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.769015    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:51.769118 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.769109    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:34:51.769173 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.769158    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.769144526 +0000 UTC m=+37.216974963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:34:51.870244 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.870212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:51.870402 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.870371    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:51.870445 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:51.870437    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.870421092 +0000 UTC m=+37.318251524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:51.956663 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.956623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf" event={"ID":"2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e","Type":"ContainerStarted","Data":"859834e0e0ec51d736e8f1093b7e9a0cc2205f65fecc8cb7c23ff02fd3b130f6"}
Sep 17 17:34:51.959234 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.959210    2562 generic.go:356] "Generic (PLEG): container finished" podID="59ba2826-88d8-472b-8c1d-2f9d06841f6d" containerID="37832f3c5abfd2b20d5e28aa0159589c071d27592947bd29aab646c6ed3fa987" exitCode=0
Sep 17 17:34:51.959385 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.959274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerDied","Data":"37832f3c5abfd2b20d5e28aa0159589c071d27592947bd29aab646c6ed3fa987"}
Sep 17 17:34:51.960382 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:51.960364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7swlh" event={"ID":"04876813-f72f-4194-8dab-62304008b825","Type":"ContainerStarted","Data":"cb77e338306bccd7cd4a83c298b6ccbe1f14441c4cc2493afa289b590530a9c7"}
Sep 17 17:34:52.722083 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.722045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:34:52.722350 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.722045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:34:52.722350 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.722045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:52.728325 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.728300    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:52.728482 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.728382    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:52.729536 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.729512    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g24m8\""
Sep 17 17:34:52.729649 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.729564    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2vz4h\""
Sep 17 17:34:52.750762 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.750741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:52.776879 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.776851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:52.776991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.776898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:52.777054 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.777038    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:52.777116 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.777101    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.777080227 +0000 UTC m=+39.224910662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:52.777576 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.777555    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:52.777663 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.777580    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:34:52.777663 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.777630    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.777613467 +0000 UTC m=+39.225443914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:34:52.878052 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.878020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:52.878219 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.878188    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:52.878285 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:52.878277    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.878256104 +0000 UTC m=+39.326086552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:52.968798 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.968665    2562 generic.go:356] "Generic (PLEG): container finished" podID="59ba2826-88d8-472b-8c1d-2f9d06841f6d" containerID="9ac343d108bad6c276f9de20d18029fcd7f1e682a0b58729eab4ba45c09f1219" exitCode=0
Sep 17 17:34:52.968798 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:52.968739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerDied","Data":"9ac343d108bad6c276f9de20d18029fcd7f1e682a0b58729eab4ba45c09f1219"}
Sep 17 17:34:53.058625 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:53.058595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7c8nf"]
Sep 17 17:34:53.064434 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:34:53.064403    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa66574_a870_4c4a_9932_e850183522b4.slice/crio-bf2b2a6ff4e02c5b3eb1af56d82b6de59a3641a0357b79727499ced7511c82b6 WatchSource:0}: Error finding container bf2b2a6ff4e02c5b3eb1af56d82b6de59a3641a0357b79727499ced7511c82b6: Status 404 returned error can't find the container with id bf2b2a6ff4e02c5b3eb1af56d82b6de59a3641a0357b79727499ced7511c82b6
Sep 17 17:34:53.976526 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:53.976480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" event={"ID":"59ba2826-88d8-472b-8c1d-2f9d06841f6d","Type":"ContainerStarted","Data":"852db3114ca1c87a50dc0d8d3b90eff681a999e1151dd9253d29d89762e737dd"}
Sep 17 17:34:53.978220 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:53.978193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7c8nf" event={"ID":"0fa66574-a870-4c4a-9932-e850183522b4","Type":"ContainerStarted","Data":"bf2b2a6ff4e02c5b3eb1af56d82b6de59a3641a0357b79727499ced7511c82b6"}
Sep 17 17:34:54.005657 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:54.005596    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wzs8n" podStartSLOduration=8.449522961 podStartE2EDuration="36.00557943s" totalImagesPullingTime="27.556056469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:54.00361594 +0000 UTC m=+38.451446420" watchObservedRunningTime="2026-09-17 17:34:54.00557943 +0000 UTC m=+38.453409885"
Sep 17 17:34:54.795380 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:54.795341    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:54.795567 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:54.795399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:54.795567 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.795557    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:54.795682 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.795590    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:54.795682 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.795612    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:34:54.795682 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.795618    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:58.795603199 +0000 UTC m=+43.243433637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:54.795682 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.795669    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:58.795650545 +0000 UTC m=+43.243480992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:34:54.896498 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:54.896454    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:54.896682 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.896632    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:54.896753 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:54.896729    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:58.8967079 +0000 UTC m=+43.344538340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:56.988727 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:56.988470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf" event={"ID":"2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e","Type":"ContainerStarted","Data":"5bb0163fbb49e39611e9262944bc9a2136f7b4ea976cecd9ae0cbca185cab6d9"}
Sep 17 17:34:56.990083 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:56.990058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7swlh" event={"ID":"04876813-f72f-4194-8dab-62304008b825","Type":"ContainerStarted","Data":"0b6433dcc07168b9b5ab3497e96a168513edf01c8d2737a5b2a809faa8aae27a"}
Sep 17 17:34:56.991550 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:56.991529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7c8nf" event={"ID":"0fa66574-a870-4c4a-9932-e850183522b4","Type":"ContainerStarted","Data":"69a60666d4d17cad64cb599c40fab7e8567b34e80ff3668e3ad5dea5df255e08"}
Sep 17 17:34:56.991674 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:56.991661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:34:57.022073 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:57.022015    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dp2xf" podStartSLOduration=29.625339872 podStartE2EDuration="34.021996737s" totalImagesPullingTime="4.396656865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:57.020513522 +0000 UTC m=+41.468343978" watchObservedRunningTime="2026-09-17 17:34:57.021996737 +0000 UTC m=+41.469827192"
Sep 17 17:34:57.043148 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:57.043090    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7c8nf" podStartSLOduration=35.960444628 podStartE2EDuration="39.043069939s" totalImagesPullingTime="3.082625311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:57.042012448 +0000 UTC m=+41.489842915" watchObservedRunningTime="2026-09-17 17:34:57.043069939 +0000 UTC m=+41.490900392"
Sep 17 17:34:57.066204 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:57.066141    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7swlh" podStartSLOduration=1.6697256980000001 podStartE2EDuration="6.066120655s" totalImagesPullingTime="4.396394957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:57.0652102 +0000 UTC m=+41.513040655" watchObservedRunningTime="2026-09-17 17:34:57.066120655 +0000 UTC m=+41.513951110"
Sep 17 17:34:58.825087 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:58.825028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:34:58.825087 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:58.825087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:34:58.825510 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.825174    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:58.825510 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.825185    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:58.825510 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.825212    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:34:58.825510 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.825235    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:06.825221372 +0000 UTC m=+51.273051807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:58.825510 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.825256    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:06.825243449 +0000 UTC m=+51.273073882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:34:58.926518 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:34:58.926483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:34:58.926662 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.926618    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:58.926703 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:34:58.926676    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:06.926660922 +0000 UTC m=+51.374491354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:06.886826 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:06.886780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:35:06.886826 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:06.886830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:35:06.887471 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.886932    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:06.887471 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.886949    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:06.887471 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.886951    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:35:06.887471 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.887008    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.886991238 +0000 UTC m=+67.334821672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:35:06.887471 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.887025    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.887018948 +0000 UTC m=+67.334849381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:06.987849 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:06.987809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:35:06.988026 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.987954    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:06.988026 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:06.988018    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.988001889 +0000 UTC m=+67.435832327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:15.952105 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:15.952077    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5l8m4"
Sep 17 17:35:22.899592 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:22.899561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:35:22.899592 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:22.899598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:35:22.900004 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:22.899692    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:22.900004 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:22.899704    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:22.900004 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:22.899722    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:35:22.900004 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:22.899758    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:54.899742293 +0000 UTC m=+99.347572726 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:22.900004 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:22.899772    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:54.899765115 +0000 UTC m=+99.347595548 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:35:23.000713 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.000687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:35:23.000867 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:23.000820    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:23.000910 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:23.000874    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:55.000860489 +0000 UTC m=+99.448690933 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:23.402853 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.402822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:35:23.403010 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.402899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:35:23.405860 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.405840    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:23.405928 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.405871    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:23.413223 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:23.413202    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:23.413283 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:23.413266    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:27.413249647 +0000 UTC m=+131.861080091 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : secret "metrics-daemon-secret" not found
Sep 17 17:35:23.416617 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.416598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c83b0a2-9af0-4601-9d20-79b3a6533dbf-original-pull-secret\") pod \"global-pull-secret-syncer-d2v8l\" (UID: \"6c83b0a2-9af0-4601-9d20-79b3a6533dbf\") " pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:35:23.635708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.635676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d2v8l"
Sep 17 17:35:23.906627 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:23.906604    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d2v8l"]
Sep 17 17:35:23.909078 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:35:23.909056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c83b0a2_9af0_4601_9d20_79b3a6533dbf.slice/crio-3f6604854e003bd17286cc2481cea4195fbc2d91b9e33465efc303d7f867ce80 WatchSource:0}: Error finding container 3f6604854e003bd17286cc2481cea4195fbc2d91b9e33465efc303d7f867ce80: Status 404 returned error can't find the container with id 3f6604854e003bd17286cc2481cea4195fbc2d91b9e33465efc303d7f867ce80
Sep 17 17:35:24.056123 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:24.056067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2v8l" event={"ID":"6c83b0a2-9af0-4601-9d20-79b3a6533dbf","Type":"ContainerStarted","Data":"3f6604854e003bd17286cc2481cea4195fbc2d91b9e33465efc303d7f867ce80"}
Sep 17 17:35:27.995987 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:27.995961    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7c8nf"
Sep 17 17:35:28.068767 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:28.068735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d2v8l" event={"ID":"6c83b0a2-9af0-4601-9d20-79b3a6533dbf","Type":"ContainerStarted","Data":"04fef1b8f79a9ee7b8727c83e0377a1b382ed9bfad7c8e5041bfaea5a800e1bd"}
Sep 17 17:35:28.087853 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:28.087803    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d2v8l" podStartSLOduration=66.519540131 podStartE2EDuration="1m10.087786549s" totalImagesPullingTime="3.568246418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:28.08728882 +0000 UTC m=+72.535119300" watchObservedRunningTime="2026-09-17 17:35:28.087786549 +0000 UTC m=+72.535617036"
Sep 17 17:35:54.912661 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:54.912585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:35:54.912661 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:54.912628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:35:54.913067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:54.912736    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:54.913067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:54.912760    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:54.913067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:54.912785    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:35:54.913067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:54.912798    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:58.912785719 +0000 UTC m=+163.360616152 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:54.913067 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:54.912841    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:58.912828633 +0000 UTC m=+163.360659070 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:35:55.014076 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:35:55.014041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:35:55.014177 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:55.014165    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:55.014225 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:35:55.014217    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:59.014203681 +0000 UTC m=+163.462034115 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:27.452108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:27.452066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:36:27.452605 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:27.452218    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:36:27.452605 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:27.452297    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs podName:fbc60926-841c-4824-872b-30a7c806ade0 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:29.452280126 +0000 UTC m=+253.900110559 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs") pod "network-metrics-daemon-rzpqd" (UID: "fbc60926-841c-4824-872b-30a7c806ade0") : secret "metrics-daemon-secret" not found
Sep 17 17:36:54.073162 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:54.073109    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw" podUID="82a1fb15-93eb-46b6-a334-5ca33f438d14"
Sep 17 17:36:54.084298 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:54.084264    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp" podUID="d916b8ba-958f-4cb4-aadc-ab6f3467ac17"
Sep 17 17:36:54.120581 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:54.120543    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bljxr" podUID="c86541f5-fa09-4a99-b838-01322d9c0cb6"
Sep 17 17:36:54.277103 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:54.277072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bljxr"
Sep 17 17:36:54.277103 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:54.277102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:36:54.277291 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:54.277210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:36:55.744408 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:55.744360    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rzpqd" podUID="fbc60926-841c-4824-872b-30a7c806ade0"
Sep 17 17:36:55.993694 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:55.993653    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"]
Sep 17 17:36:55.996457 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:55.996419    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:55.999679 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:55.999659    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:56.000766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.000743    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5tx9p\""
Sep 17 17:36:56.000766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.000748    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:56.002095 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.002077    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fn7x9"]
Sep 17 17:36:56.004687 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.004670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.006446 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.006430    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:36:56.008998 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.008983    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:36:56.009069 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.009055    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:36:56.009225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.009205    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:56.009991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.009847    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lgnvn\""
Sep 17 17:36:56.010093 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.010056    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:56.017992 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.017971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"]
Sep 17 17:36:56.020001 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.019981    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fn7x9"]
Sep 17 17:36:56.023783 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.023762    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:36:56.085364 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.085324    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"]
Sep 17 17:36:56.088165 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.088149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.091398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.091377    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:56.091683 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.091667    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:36:56.091851 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.091838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:36:56.092274 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.092256    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vrdtr\""
Sep 17 17:36:56.092815 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.092801    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:56.099896 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.099877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"]
Sep 17 17:36:56.155343 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.155299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7de942b3-1609-4be9-9632-b39a80a5d381-trusted-ca\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.155504 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.155356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rnj\" (UniqueName: \"kubernetes.io/projected/7de942b3-1609-4be9-9632-b39a80a5d381-kube-api-access-r7rnj\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.155504 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.155436    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:56.155576 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.155503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7de942b3-1609-4be9-9632-b39a80a5d381-serving-cert\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.155576 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.155559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n6sv\" (UniqueName: \"kubernetes.io/projected/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-kube-api-access-8n6sv\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:56.155650 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.155587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7de942b3-1609-4be9-9632-b39a80a5d381-config\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.256513 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7de942b3-1609-4be9-9632-b39a80a5d381-config\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.256513 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lzh\" (UniqueName: \"kubernetes.io/projected/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-kube-api-access-q9lzh\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.256739 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.256739 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.256739 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7de942b3-1609-4be9-9632-b39a80a5d381-trusted-ca\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.256739 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rnj\" (UniqueName: \"kubernetes.io/projected/7de942b3-1609-4be9-9632-b39a80a5d381-kube-api-access-r7rnj\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.256739 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:56.256739 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7de942b3-1609-4be9-9632-b39a80a5d381-serving-cert\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.257049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.256750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n6sv\" (UniqueName: \"kubernetes.io/projected/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-kube-api-access-8n6sv\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:56.257049 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:56.256820    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:56.257049 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:56.256888    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls podName:62c9eba5-51e7-4592-8c4f-7f13bcfceeff nodeName:}" failed. No retries permitted until 2026-09-17 17:36:56.756869706 +0000 UTC m=+161.204700157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-klft8" (UID: "62c9eba5-51e7-4592-8c4f-7f13bcfceeff") : secret "samples-operator-tls" not found
Sep 17 17:36:56.257259 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.257238    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7de942b3-1609-4be9-9632-b39a80a5d381-config\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.257417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.257400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7de942b3-1609-4be9-9632-b39a80a5d381-trusted-ca\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.258932 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.258911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7de942b3-1609-4be9-9632-b39a80a5d381-serving-cert\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.267960 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.267937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n6sv\" (UniqueName: \"kubernetes.io/projected/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-kube-api-access-8n6sv\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:56.270477 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.270458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rnj\" (UniqueName: \"kubernetes.io/projected/7de942b3-1609-4be9-9632-b39a80a5d381-kube-api-access-r7rnj\") pod \"console-operator-fd4bb46c-fn7x9\" (UID: \"7de942b3-1609-4be9-9632-b39a80a5d381\") " pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.312112 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.312068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:36:56.357301 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.357270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lzh\" (UniqueName: \"kubernetes.io/projected/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-kube-api-access-q9lzh\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.357433 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.357315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.357433 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.357365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.358073 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.358046    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.360322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.360301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.372662 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.372640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lzh\" (UniqueName: \"kubernetes.io/projected/8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13-kube-api-access-q9lzh\") pod \"kube-storage-version-migrator-operator-55889f7b47-bjwcr\" (UID: \"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.396588 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.396561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"
Sep 17 17:36:56.583044 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.583014    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-fn7x9"]
Sep 17 17:36:56.586122 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:36:56.586097    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de942b3_1609_4be9_9632_b39a80a5d381.slice/crio-c006e1e6fec81c3ba2de98088944638f73b5a22ad65763c8068e80e115815a2e WatchSource:0}: Error finding container c006e1e6fec81c3ba2de98088944638f73b5a22ad65763c8068e80e115815a2e: Status 404 returned error can't find the container with id c006e1e6fec81c3ba2de98088944638f73b5a22ad65763c8068e80e115815a2e
Sep 17 17:36:56.664617 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.664591    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr"]
Sep 17 17:36:56.667564 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:36:56.667542    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbb2901_fd04_4f6f_a6c6_65b4f61dbb13.slice/crio-afb91898aeca46f802ec62555dc520f4d70bd66fad9f99f670388f0405a3aed7 WatchSource:0}: Error finding container afb91898aeca46f802ec62555dc520f4d70bd66fad9f99f670388f0405a3aed7: Status 404 returned error can't find the container with id afb91898aeca46f802ec62555dc520f4d70bd66fad9f99f670388f0405a3aed7
Sep 17 17:36:56.760426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:56.760386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:56.760815 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:56.760534    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:56.760815 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:56.760608    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls podName:62c9eba5-51e7-4592-8c4f-7f13bcfceeff nodeName:}" failed. No retries permitted until 2026-09-17 17:36:57.760592499 +0000 UTC m=+162.208422935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-klft8" (UID: "62c9eba5-51e7-4592-8c4f-7f13bcfceeff") : secret "samples-operator-tls" not found
Sep 17 17:36:57.285519 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:57.285479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr" event={"ID":"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13","Type":"ContainerStarted","Data":"afb91898aeca46f802ec62555dc520f4d70bd66fad9f99f670388f0405a3aed7"}
Sep 17 17:36:57.286564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:57.286544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" event={"ID":"7de942b3-1609-4be9-9632-b39a80a5d381","Type":"ContainerStarted","Data":"c006e1e6fec81c3ba2de98088944638f73b5a22ad65763c8068e80e115815a2e"}
Sep 17 17:36:57.768972 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:57.768918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:57.769373 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:57.769076    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:57.769373 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:57.769136    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls podName:62c9eba5-51e7-4592-8c4f-7f13bcfceeff nodeName:}" failed. No retries permitted until 2026-09-17 17:36:59.769119799 +0000 UTC m=+164.216950240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-klft8" (UID: "62c9eba5-51e7-4592-8c4f-7f13bcfceeff") : secret "samples-operator-tls" not found
Sep 17 17:36:58.979654 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:58.979623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") pod \"image-registry-6d74d8ff8d-fxlfw\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") " pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:36:58.980099 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:58.979677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:36:58.980099 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:58.979779    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:58.980099 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:58.979801    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d74d8ff8d-fxlfw: secret "image-registry-tls" not found
Sep 17 17:36:58.980099 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:58.979799    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:36:58.980099 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:58.979855    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls podName:82a1fb15-93eb-46b6-a334-5ca33f438d14 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:00.979838415 +0000 UTC m=+285.427668848 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls") pod "image-registry-6d74d8ff8d-fxlfw" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14") : secret "image-registry-tls" not found
Sep 17 17:36:58.980099 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:58.979868    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert podName:d916b8ba-958f-4cb4-aadc-ab6f3467ac17 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:00.979861761 +0000 UTC m=+285.427692195 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-8lqdp" (UID: "d916b8ba-958f-4cb4-aadc-ab6f3467ac17") : secret "networking-console-plugin-cert" not found
Sep 17 17:36:59.080768 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:59.080741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:36:59.080861 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:59.080840    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:59.080904 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:59.080897    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls podName:c86541f5-fa09-4a99-b838-01322d9c0cb6 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:01.080883878 +0000 UTC m=+285.528714315 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls") pod "dns-default-bljxr" (UID: "c86541f5-fa09-4a99-b838-01322d9c0cb6") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:59.294218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:59.294104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr" event={"ID":"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13","Type":"ContainerStarted","Data":"43359f29952efafb93b3f097334087d09cf0a68c352bee500343da3afad9c68f"}
Sep 17 17:36:59.327044 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:59.326959    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr" podStartSLOduration=1.4536782050000001 podStartE2EDuration="3.326942795s" totalImagesPullingTime="1.87326459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:59.326677349 +0000 UTC m=+163.774507803" watchObservedRunningTime="2026-09-17 17:36:59.326942795 +0000 UTC m=+163.774773250"
Sep 17 17:36:59.785104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:36:59.785068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:36:59.785263 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:59.785215    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:59.785304 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:36:59.785279    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls podName:62c9eba5-51e7-4592-8c4f-7f13bcfceeff nodeName:}" failed. No retries permitted until 2026-09-17 17:37:03.785262554 +0000 UTC m=+168.233092991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-klft8" (UID: "62c9eba5-51e7-4592-8c4f-7f13bcfceeff") : secret "samples-operator-tls" not found
Sep 17 17:37:00.944488 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.944455    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-54hqg"]
Sep 17 17:37:00.947430 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.947412    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:00.950042 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.950021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:37:00.950199 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.950027    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:37:00.950199 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.950027    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:37:00.950360 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.950346    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:37:00.951185 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.951170    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9vw5\""
Sep 17 17:37:00.964356 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:00.964168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54hqg"]
Sep 17 17:37:01.096289 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.096256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34f19014-b54a-417b-a27d-8211a23fb72e-data-volume\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.096289 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.096288    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjx8\" (UniqueName: \"kubernetes.io/projected/34f19014-b54a-417b-a27d-8211a23fb72e-kube-api-access-5pjx8\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.096500 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.096410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34f19014-b54a-417b-a27d-8211a23fb72e-crio-socket\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.096500 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.096467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.096500 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.096486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.197847 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.197757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34f19014-b54a-417b-a27d-8211a23fb72e-data-volume\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.197847 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.197803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjx8\" (UniqueName: \"kubernetes.io/projected/34f19014-b54a-417b-a27d-8211a23fb72e-kube-api-access-5pjx8\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.197847 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.197849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34f19014-b54a-417b-a27d-8211a23fb72e-crio-socket\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.198068 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.197886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.198068 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.197911    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.198068 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.197979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34f19014-b54a-417b-a27d-8211a23fb72e-crio-socket\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.198068 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.198003    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.198068 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.198040    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.198215 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.198088    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:01.698066649 +0000 UTC m=+166.145897106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.198215 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.198108    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:01.698097591 +0000 UTC m=+166.145928026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.198215 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.198150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34f19014-b54a-417b-a27d-8211a23fb72e-data-volume\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.210581 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.210562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjx8\" (UniqueName: \"kubernetes.io/projected/34f19014-b54a-417b-a27d-8211a23fb72e-kube-api-access-5pjx8\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.701499 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.701458    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.701703 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:01.701509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:01.701703 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.701622    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.701703 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.701683    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:02.701656366 +0000 UTC m=+167.149486821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.701703 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.701701    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.701910 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:01.701751    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:02.701741146 +0000 UTC m=+167.149571579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:02.303167 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:02.303101    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/0.log"
Sep 17 17:37:02.303167 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:02.303137    2562 generic.go:356] "Generic (PLEG): container finished" podID="7de942b3-1609-4be9-9632-b39a80a5d381" containerID="0a877bf1d219185cec0b704b79c6885f1ebd7cd0db8e9467f7de0d3508fffc62" exitCode=255
Sep 17 17:37:02.303550 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:02.303189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" event={"ID":"7de942b3-1609-4be9-9632-b39a80a5d381","Type":"ContainerDied","Data":"0a877bf1d219185cec0b704b79c6885f1ebd7cd0db8e9467f7de0d3508fffc62"}
Sep 17 17:37:02.303550 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:02.303448    2562 scope.go:117] "RemoveContainer" containerID="0a877bf1d219185cec0b704b79c6885f1ebd7cd0db8e9467f7de0d3508fffc62"
Sep 17 17:37:02.708583 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:02.708550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:02.708583 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:02.708583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:02.708778 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:02.708664    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:02.708778 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:02.708682    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:02.708778 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:02.708723    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:04.708708666 +0000 UTC m=+169.156539130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:02.708778 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:02.708736    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:04.708730383 +0000 UTC m=+169.156560815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:03.241170 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.241137    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pztqd"]
Sep 17 17:37:03.244178 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.244164    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.246965 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.246946    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:37:03.247243 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.247226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:37:03.248426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.248411    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:37:03.248505 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.248446    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:37:03.248569 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.248504    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7qpj6\""
Sep 17 17:37:03.256107 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.256086    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pztqd"]
Sep 17 17:37:03.307531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.307506    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/1.log"
Sep 17 17:37:03.307920 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.307907    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/0.log"
Sep 17 17:37:03.307964 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.307939    2562 generic.go:356] "Generic (PLEG): container finished" podID="7de942b3-1609-4be9-9632-b39a80a5d381" containerID="139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf" exitCode=255
Sep 17 17:37:03.307998 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.307980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" event={"ID":"7de942b3-1609-4be9-9632-b39a80a5d381","Type":"ContainerDied","Data":"139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf"}
Sep 17 17:37:03.308030 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.308018    2562 scope.go:117] "RemoveContainer" containerID="0a877bf1d219185cec0b704b79c6885f1ebd7cd0db8e9467f7de0d3508fffc62"
Sep 17 17:37:03.308262 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.308246    2562 scope.go:117] "RemoveContainer" containerID="139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf"
Sep 17 17:37:03.308453 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:03.308435    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fn7x9_openshift-console-operator(7de942b3-1609-4be9-9632-b39a80a5d381)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podUID="7de942b3-1609-4be9-9632-b39a80a5d381"
Sep 17 17:37:03.414797 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.414762    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf874\" (UniqueName: \"kubernetes.io/projected/fcefdf70-1726-4477-8c04-c26cf8a672bf-kube-api-access-jf874\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.414956 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.414926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcefdf70-1726-4477-8c04-c26cf8a672bf-signing-key\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.415048 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.415027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcefdf70-1726-4477-8c04-c26cf8a672bf-signing-cabundle\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.516193 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.516131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcefdf70-1726-4477-8c04-c26cf8a672bf-signing-key\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.516193 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.516187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcefdf70-1726-4477-8c04-c26cf8a672bf-signing-cabundle\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.516300 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.516248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf874\" (UniqueName: \"kubernetes.io/projected/fcefdf70-1726-4477-8c04-c26cf8a672bf-kube-api-access-jf874\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.516931 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.516898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcefdf70-1726-4477-8c04-c26cf8a672bf-signing-cabundle\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.518531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.518514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcefdf70-1726-4477-8c04-c26cf8a672bf-signing-key\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.530840 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.530818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf874\" (UniqueName: \"kubernetes.io/projected/fcefdf70-1726-4477-8c04-c26cf8a672bf-kube-api-access-jf874\") pod \"service-ca-55f6fc7b86-pztqd\" (UID: \"fcefdf70-1726-4477-8c04-c26cf8a672bf\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.552692 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.552672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd"
Sep 17 17:37:03.819555 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.819515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:37:03.819708 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:03.819634    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:37:03.819708 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:03.819687    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls podName:62c9eba5-51e7-4592-8c4f-7f13bcfceeff nodeName:}" failed. No retries permitted until 2026-09-17 17:37:11.819673457 +0000 UTC m=+176.267503890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-klft8" (UID: "62c9eba5-51e7-4592-8c4f-7f13bcfceeff") : secret "samples-operator-tls" not found
Sep 17 17:37:03.832194 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.832167    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pztqd"]
Sep 17 17:37:03.835131 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:03.835102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcefdf70_1726_4477_8c04_c26cf8a672bf.slice/crio-cc6ed508ee9d7cfe5d067d16407f9b28ce3d9f7f385c85144af83931cca33b45 WatchSource:0}: Error finding container cc6ed508ee9d7cfe5d067d16407f9b28ce3d9f7f385c85144af83931cca33b45: Status 404 returned error can't find the container with id cc6ed508ee9d7cfe5d067d16407f9b28ce3d9f7f385c85144af83931cca33b45
Sep 17 17:37:03.920637 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:03.920603    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mwgsw_6ebdeed2-06ee-41de-b637-0ae050cf8c63/dns-node-resolver/0.log"
Sep 17 17:37:04.311976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:04.311937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd" event={"ID":"fcefdf70-1726-4477-8c04-c26cf8a672bf","Type":"ContainerStarted","Data":"cc6ed508ee9d7cfe5d067d16407f9b28ce3d9f7f385c85144af83931cca33b45"}
Sep 17 17:37:04.313419 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:04.313395    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/1.log"
Sep 17 17:37:04.313726 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:04.313711    2562 scope.go:117] "RemoveContainer" containerID="139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf"
Sep 17 17:37:04.313865 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:04.313848    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fn7x9_openshift-console-operator(7de942b3-1609-4be9-9632-b39a80a5d381)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podUID="7de942b3-1609-4be9-9632-b39a80a5d381"
Sep 17 17:37:04.715591 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:04.715510    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j9mf9_a9557f36-6cf8-46fb-b997-9f9ade8915be/node-ca/0.log"
Sep 17 17:37:04.725903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:04.725879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:04.726038 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:04.725917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:04.726038 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:04.725995    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:04.726143 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:04.726077    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:08.726058156 +0000 UTC m=+173.173888608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:04.726143 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:04.726087    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:04.726143 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:04.726126    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:08.726115022 +0000 UTC m=+173.173945470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:05.516402 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:05.516368    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7swlh_04876813-f72f-4194-8dab-62304008b825/serve-healthcheck-canary/0.log"
Sep 17 17:37:06.312299 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:06.312261    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:37:06.312475 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:06.312315    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:37:06.312655 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:06.312641    2562 scope.go:117] "RemoveContainer" containerID="139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf"
Sep 17 17:37:06.312798 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:06.312781    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-fn7x9_openshift-console-operator(7de942b3-1609-4be9-9632-b39a80a5d381)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podUID="7de942b3-1609-4be9-9632-b39a80a5d381"
Sep 17 17:37:06.321197 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:06.321166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd" event={"ID":"fcefdf70-1726-4477-8c04-c26cf8a672bf","Type":"ContainerStarted","Data":"871c3582be0a44a3541fcf33e16c1aa043f87bc91fc97cfe38a0f2848fd6f3a4"}
Sep 17 17:37:06.325721 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:06.325697    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-bjwcr_8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13/kube-storage-version-migrator-operator/0.log"
Sep 17 17:37:08.761113 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:08.761074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:08.761113 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:08.761116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:08.761547 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:08.761201    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:08.761547 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:08.761232    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:08.761547 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:08.761270    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:16.761253486 +0000 UTC m=+181.209083924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:08.761547 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:08.761283    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:16.761277686 +0000 UTC m=+181.209108118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:10.722123 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:10.722083    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:37:11.885139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:11.885105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:37:11.887563 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:11.887538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62c9eba5-51e7-4592-8c4f-7f13bcfceeff-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-klft8\" (UID: \"62c9eba5-51e7-4592-8c4f-7f13bcfceeff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:37:11.905322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:11.905297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"
Sep 17 17:37:12.184124 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:12.184017    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-pztqd" podStartSLOduration=7.376350208 podStartE2EDuration="9.183993281s" totalImagesPullingTime="1.807643073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:06.347878091 +0000 UTC m=+170.795708573" watchObservedRunningTime="2026-09-17 17:37:12.183993281 +0000 UTC m=+176.631823740"
Sep 17 17:37:12.184444 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:12.184424    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8"]
Sep 17 17:37:12.339033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:12.338996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8" event={"ID":"62c9eba5-51e7-4592-8c4f-7f13bcfceeff","Type":"ContainerStarted","Data":"7f8007f39127299b08124e2b6944da37231ee1c7bb4da0948f0acf2ee4b2e92c"}
Sep 17 17:37:14.348968 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:14.348860    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8" event={"ID":"62c9eba5-51e7-4592-8c4f-7f13bcfceeff","Type":"ContainerStarted","Data":"47b8ed570953d75966aebe51459c1e953cc57393b1cbd25bdaaf0153cf8c13f2"}
Sep 17 17:37:14.348968 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:14.348932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8" event={"ID":"62c9eba5-51e7-4592-8c4f-7f13bcfceeff","Type":"ContainerStarted","Data":"2e92d307e8709097efa311ce279176e0e6f5d36d5619c8370b1dc687da08b7fe"}
Sep 17 17:37:14.374959 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:14.374915    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-klft8" podStartSLOduration=17.505182812 podStartE2EDuration="19.374899216s" totalImagesPullingTime="1.869716404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:14.373360949 +0000 UTC m=+178.821191431" watchObservedRunningTime="2026-09-17 17:37:14.374899216 +0000 UTC m=+178.822729671"
Sep 17 17:37:16.825557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:16.825521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:16.825928 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:16.825568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:16.825928 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:16.825743    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:16.825928 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:16.825806    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls podName:34f19014-b54a-417b-a27d-8211a23fb72e nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.825789509 +0000 UTC m=+197.273619948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-54hqg" (UID: "34f19014-b54a-417b-a27d-8211a23fb72e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:16.826058 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:16.826040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34f19014-b54a-417b-a27d-8211a23fb72e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:19.721671 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:19.721642    2562 scope.go:117] "RemoveContainer" containerID="139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf"
Sep 17 17:37:20.369156 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:20.369131    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:37:20.369566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:20.369548    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/1.log"
Sep 17 17:37:20.369671 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:20.369586    2562 generic.go:356] "Generic (PLEG): container finished" podID="7de942b3-1609-4be9-9632-b39a80a5d381" containerID="1b5067628be3abfdc4719ef724002348868fa9f1505d6b38ca98f52416eed23d" exitCode=255
Sep 17 17:37:20.369671 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:20.369660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" event={"ID":"7de942b3-1609-4be9-9632-b39a80a5d381","Type":"ContainerDied","Data":"1b5067628be3abfdc4719ef724002348868fa9f1505d6b38ca98f52416eed23d"}
Sep 17 17:37:20.369766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:20.369720    2562 scope.go:117] "RemoveContainer" containerID="139bab0bc3410bdb03cb0d9e339ab78bf6d8b582c14f7a16b500a04276feb2cf"
Sep 17 17:37:20.369999 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:20.369970    2562 scope.go:117] "RemoveContainer" containerID="1b5067628be3abfdc4719ef724002348868fa9f1505d6b38ca98f52416eed23d"
Sep 17 17:37:20.370139 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:20.370122    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-fn7x9_openshift-console-operator(7de942b3-1609-4be9-9632-b39a80a5d381)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podUID="7de942b3-1609-4be9-9632-b39a80a5d381"
Sep 17 17:37:21.374520 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:21.374495    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:37:26.312921 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:26.312884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:37:26.313288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:26.312937    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:37:26.313288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:26.313222    2562 scope.go:117] "RemoveContainer" containerID="1b5067628be3abfdc4719ef724002348868fa9f1505d6b38ca98f52416eed23d"
Sep 17 17:37:26.313411 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:26.313391    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-fn7x9_openshift-console-operator(7de942b3-1609-4be9-9632-b39a80a5d381)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podUID="7de942b3-1609-4be9-9632-b39a80a5d381"
Sep 17 17:37:31.338346 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.338290    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"]
Sep 17 17:37:31.340862 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.340844    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:31.345259 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.345238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:37:31.345880 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.345864    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ps7qw\""
Sep 17 17:37:31.354941 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.354915    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"]
Sep 17 17:37:31.434506 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.434461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4570300d-7d11-49ce-8a0a-56523a07b61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-92bpv\" (UID: \"4570300d-7d11-49ce-8a0a-56523a07b61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:31.535465 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:31.535430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4570300d-7d11-49ce-8a0a-56523a07b61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-92bpv\" (UID: \"4570300d-7d11-49ce-8a0a-56523a07b61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:31.535604 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:31.535581    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 17:37:31.535685 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:31.535675    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4570300d-7d11-49ce-8a0a-56523a07b61a-tls-certificates podName:4570300d-7d11-49ce-8a0a-56523a07b61a nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.035658067 +0000 UTC m=+196.483488504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4570300d-7d11-49ce-8a0a-56523a07b61a-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-92bpv" (UID: "4570300d-7d11-49ce-8a0a-56523a07b61a") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 17:37:32.040800 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:32.040751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4570300d-7d11-49ce-8a0a-56523a07b61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-92bpv\" (UID: \"4570300d-7d11-49ce-8a0a-56523a07b61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:32.043140 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:32.043118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4570300d-7d11-49ce-8a0a-56523a07b61a-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-92bpv\" (UID: \"4570300d-7d11-49ce-8a0a-56523a07b61a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:32.250954 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:32.250903    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:32.535013 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:32.534862    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"]
Sep 17 17:37:32.537262 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:32.537233    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4570300d_7d11_49ce_8a0a_56523a07b61a.slice/crio-ff352c4c41c9bf57275293b929cfa6fefe7d2bf491f60cca15277be309225f42 WatchSource:0}: Error finding container ff352c4c41c9bf57275293b929cfa6fefe7d2bf491f60cca15277be309225f42: Status 404 returned error can't find the container with id ff352c4c41c9bf57275293b929cfa6fefe7d2bf491f60cca15277be309225f42
Sep 17 17:37:32.846631 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:32.846590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:32.848923 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:32.848895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34f19014-b54a-417b-a27d-8211a23fb72e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-54hqg\" (UID: \"34f19014-b54a-417b-a27d-8211a23fb72e\") " pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:33.059658 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:33.059615    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9vw5\""
Sep 17 17:37:33.067070 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:33.067038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-54hqg"
Sep 17 17:37:33.347875 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:33.347846    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-54hqg"]
Sep 17 17:37:33.412132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:33.412066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv" event={"ID":"4570300d-7d11-49ce-8a0a-56523a07b61a","Type":"ContainerStarted","Data":"ff352c4c41c9bf57275293b929cfa6fefe7d2bf491f60cca15277be309225f42"}
Sep 17 17:37:33.500056 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:33.500028    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f19014_b54a_417b_a27d_8211a23fb72e.slice/crio-94b5eb0afeeaf1c956a5107d6b74725248632b108d06e056876f1fafe68dff20 WatchSource:0}: Error finding container 94b5eb0afeeaf1c956a5107d6b74725248632b108d06e056876f1fafe68dff20: Status 404 returned error can't find the container with id 94b5eb0afeeaf1c956a5107d6b74725248632b108d06e056876f1fafe68dff20
Sep 17 17:37:34.417057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.416981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54hqg" event={"ID":"34f19014-b54a-417b-a27d-8211a23fb72e","Type":"ContainerStarted","Data":"9c2bf004684221bc8c7552b5bd585dcc5c08db980d6909d9abd5462cd8bb0880"}
Sep 17 17:37:34.417057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.417037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54hqg" event={"ID":"34f19014-b54a-417b-a27d-8211a23fb72e","Type":"ContainerStarted","Data":"6bf873e83f3c510687ae18e53a2836517e2bed1585ffd93b985b33458a7edc87"}
Sep 17 17:37:34.417460 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.417061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54hqg" event={"ID":"34f19014-b54a-417b-a27d-8211a23fb72e","Type":"ContainerStarted","Data":"94b5eb0afeeaf1c956a5107d6b74725248632b108d06e056876f1fafe68dff20"}
Sep 17 17:37:34.418475 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.418450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv" event={"ID":"4570300d-7d11-49ce-8a0a-56523a07b61a","Type":"ContainerStarted","Data":"011137f0fe00b244ded4680b9ffc027daa3626b98ab7c40f445935034638ce0a"}
Sep 17 17:37:34.418752 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.418737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:34.422937 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.422914    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv"
Sep 17 17:37:34.443999 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:34.443964    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-92bpv" podStartSLOduration=2.446050771 podStartE2EDuration="3.443953446s" totalImagesPullingTime="997.902675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:34.443173697 +0000 UTC m=+198.891004165" watchObservedRunningTime="2026-09-17 17:37:34.443953446 +0000 UTC m=+198.891783901"
Sep 17 17:37:35.398396 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.398359    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"]
Sep 17 17:37:35.401784 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.401766    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.404306 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.404277    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:37:35.405290 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.405257    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:37:35.405290 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.405271    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:37:35.405470 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.405339    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:37:35.405470 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.405442    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:37:35.405555 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.405494    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wkrxb\""
Sep 17 17:37:35.417748 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.417727    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"]
Sep 17 17:37:35.467351 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.467313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8eb0f76-cca0-4524-aba9-514e8edf0534-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.467469 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.467400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.467533 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.467465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.467585 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.467534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5tc\" (UniqueName: \"kubernetes.io/projected/b8eb0f76-cca0-4524-aba9-514e8edf0534-kube-api-access-2x5tc\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.568413 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.568387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.568536 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.568431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5tc\" (UniqueName: \"kubernetes.io/projected/b8eb0f76-cca0-4524-aba9-514e8edf0534-kube-api-access-2x5tc\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.568536 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.568477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8eb0f76-cca0-4524-aba9-514e8edf0534-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.568536 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.568507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.568695 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:35.568539    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 17:37:35.568695 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:35.568601    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-tls podName:b8eb0f76-cca0-4524-aba9-514e8edf0534 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:36.068581394 +0000 UTC m=+200.516411836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-cpfzx" (UID: "b8eb0f76-cca0-4524-aba9-514e8edf0534") : secret "prometheus-operator-tls" not found
Sep 17 17:37:35.569126 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.569109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8eb0f76-cca0-4524-aba9-514e8edf0534-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.570605 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.570588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:35.581286 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:35.581263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5tc\" (UniqueName: \"kubernetes.io/projected/b8eb0f76-cca0-4524-aba9-514e8edf0534-kube-api-access-2x5tc\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:36.071489 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:36.071445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:36.073934 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:36.073900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8eb0f76-cca0-4524-aba9-514e8edf0534-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-cpfzx\" (UID: \"b8eb0f76-cca0-4524-aba9-514e8edf0534\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:36.311077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:36.311046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"
Sep 17 17:37:36.426555 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:36.426520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-54hqg" event={"ID":"34f19014-b54a-417b-a27d-8211a23fb72e","Type":"ContainerStarted","Data":"deda7a00549b751af52e20f72add4214eafdbdbb3bdafa65a1942fcd67343804"}
Sep 17 17:37:36.450305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:36.450260    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-54hqg" podStartSLOduration=34.604392659 podStartE2EDuration="36.450245529s" totalImagesPullingTime="1.84585287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:36.448767644 +0000 UTC m=+200.896598120" watchObservedRunningTime="2026-09-17 17:37:36.450245529 +0000 UTC m=+200.898075984"
Sep 17 17:37:36.580444 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:36.580418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx"]
Sep 17 17:37:36.583158 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:36.583130    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8eb0f76_cca0_4524_aba9_514e8edf0534.slice/crio-228d78efd7d5199c0f0460b9024c886d4980f7bf596e19d8869396f13bb5da4f WatchSource:0}: Error finding container 228d78efd7d5199c0f0460b9024c886d4980f7bf596e19d8869396f13bb5da4f: Status 404 returned error can't find the container with id 228d78efd7d5199c0f0460b9024c886d4980f7bf596e19d8869396f13bb5da4f
Sep 17 17:37:37.432078 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:37.432044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx" event={"ID":"b8eb0f76-cca0-4524-aba9-514e8edf0534","Type":"ContainerStarted","Data":"228d78efd7d5199c0f0460b9024c886d4980f7bf596e19d8869396f13bb5da4f"}
Sep 17 17:37:37.723550 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:37.723470    2562 scope.go:117] "RemoveContainer" containerID="1b5067628be3abfdc4719ef724002348868fa9f1505d6b38ca98f52416eed23d"
Sep 17 17:37:37.723701 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:37.723631    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-fn7x9_openshift-console-operator(7de942b3-1609-4be9-9632-b39a80a5d381)\"" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podUID="7de942b3-1609-4be9-9632-b39a80a5d381"
Sep 17 17:37:38.440608 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:38.440576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx" event={"ID":"b8eb0f76-cca0-4524-aba9-514e8edf0534","Type":"ContainerStarted","Data":"d5c0691b1de327c1d51a5d7271c25d03a059fdf979d7e6a554b419e203335b14"}
Sep 17 17:37:38.440947 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:38.440651    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx" event={"ID":"b8eb0f76-cca0-4524-aba9-514e8edf0534","Type":"ContainerStarted","Data":"ef728b91469badf784d93a2b3901dedd431678d6288667d5f23cf2d8baf4aed2"}
Sep 17 17:37:38.460246 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:38.460202    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-cpfzx" podStartSLOduration=2.221553509 podStartE2EDuration="3.46019079s" totalImagesPullingTime="1.238637281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:38.459403211 +0000 UTC m=+202.907233685" watchObservedRunningTime="2026-09-17 17:37:38.46019079 +0000 UTC m=+202.908021245"
Sep 17 17:37:40.775030 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.774989    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"]
Sep 17 17:37:40.781393 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.781371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:40.784249 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.784227    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:37:40.784399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.784279    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:37:40.784562 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.784537    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fztzd\""
Sep 17 17:37:40.794346 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.794316    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"]
Sep 17 17:37:40.824418 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.824394    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"]
Sep 17 17:37:40.828449 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.828428    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tz26h"]
Sep 17 17:37:40.828571 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.828548    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.831564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.831542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:37:40.831848 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.831828    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vp8tq\""
Sep 17 17:37:40.831940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.831892    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:37:40.832099 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.832083    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:37:40.832418 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.832403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.835114 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.835096    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:37:40.835351 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.835228    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sg5vz\""
Sep 17 17:37:40.835351 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.835119    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:37:40.835510 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.835467    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:37:40.841142 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.841117    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"]
Sep 17 17:37:40.909738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d72f17-d779-41b8-8450-0e6a76f53444-metrics-client-ca\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.909915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902c01da-6632-47c1-80c1-c434756bdd6d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.909915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:40.909915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-accelerators-collector-config\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909911    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.910049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvbc\" (UniqueName: \"kubernetes.io/projected/902c01da-6632-47c1-80c1-c434756bdd6d-kube-api-access-6kvbc\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.910049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.909982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/902c01da-6632-47c1-80c1-c434756bdd6d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.910049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.910255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-textfile\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-sys\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-wtmp\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910207    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:40.910255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc892\" (UniqueName: \"kubernetes.io/projected/1a0927c0-029c-459a-ada7-92c1ebc06c63-kube-api-access-wc892\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:40.910531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-root\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a0927c0-029c-459a-ada7-92c1ebc06c63-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:40.910531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-tls\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:40.910531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:40.910531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:40.910460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrg6j\" (UniqueName: \"kubernetes.io/projected/a0d72f17-d779-41b8-8450-0e6a76f53444-kube-api-access-zrg6j\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011537 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d72f17-d779-41b8-8450-0e6a76f53444-metrics-client-ca\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902c01da-6632-47c1-80c1-c434756bdd6d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.011708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.011708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-accelerators-collector-config\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.011708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011685    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvbc\" (UniqueName: \"kubernetes.io/projected/902c01da-6632-47c1-80c1-c434756bdd6d-kube-api-access-6kvbc\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/902c01da-6632-47c1-80c1-c434756bdd6d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-textfile\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-sys\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-wtmp\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc892\" (UniqueName: \"kubernetes.io/projected/1a0927c0-029c-459a-ada7-92c1ebc06c63-kube-api-access-wc892\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-root\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a0927c0-029c-459a-ada7-92c1ebc06c63-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.011962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.011966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-tls\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:41.011974    2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:41.012052    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-tls podName:1a0927c0-029c-459a-ada7-92c1ebc06c63 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:41.512033194 +0000 UTC m=+205.959863631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-stqjh" (UID: "1a0927c0-029c-459a-ada7-92c1ebc06c63") : secret "openshift-state-metrics-tls" not found
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrg6j\" (UniqueName: \"kubernetes.io/projected/a0d72f17-d779-41b8-8450-0e6a76f53444-kube-api-access-zrg6j\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d72f17-d779-41b8-8450-0e6a76f53444-metrics-client-ca\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/902c01da-6632-47c1-80c1-c434756bdd6d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.012435 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012406    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-sys\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.012962 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.012936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.013074 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.013052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-wtmp\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.013175 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:41.013142    2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:37:41.013237 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:41.013189    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-tls podName:902c01da-6632-47c1-80c1-c434756bdd6d nodeName:}" failed. No retries permitted until 2026-09-17 17:37:41.513175017 +0000 UTC m=+205.961005455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-5mc2q" (UID: "902c01da-6632-47c1-80c1-c434756bdd6d") : secret "kube-state-metrics-tls" not found
Sep 17 17:37:41.013305 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:41.013264    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:37:41.013305 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:41.013294    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-tls podName:a0d72f17-d779-41b8-8450-0e6a76f53444 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:41.513283792 +0000 UTC m=+205.961114231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-tls") pod "node-exporter-tz26h" (UID: "a0d72f17-d779-41b8-8450-0e6a76f53444") : secret "node-exporter-tls" not found
Sep 17 17:37:41.013431 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.013351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0d72f17-d779-41b8-8450-0e6a76f53444-root\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.013530 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.013512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-textfile\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.013627 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.013585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a0927c0-029c-459a-ada7-92c1ebc06c63-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.013834 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.013798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/902c01da-6632-47c1-80c1-c434756bdd6d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.014289 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.014270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-accelerators-collector-config\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.014927 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.014901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.015345 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.015313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.016123 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.016102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.032407 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.031711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc892\" (UniqueName: \"kubernetes.io/projected/1a0927c0-029c-459a-ada7-92c1ebc06c63-kube-api-access-wc892\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.034743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.034702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrg6j\" (UniqueName: \"kubernetes.io/projected/a0d72f17-d779-41b8-8450-0e6a76f53444-kube-api-access-zrg6j\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.040339 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.040304    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvbc\" (UniqueName: \"kubernetes.io/projected/902c01da-6632-47c1-80c1-c434756bdd6d-kube-api-access-6kvbc\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.517766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.517731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.517973 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.517801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.517973 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.517827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-tls\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.520206 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.520177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0d72f17-d779-41b8-8450-0e6a76f53444-node-exporter-tls\") pod \"node-exporter-tz26h\" (UID: \"a0d72f17-d779-41b8-8450-0e6a76f53444\") " pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.520356 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.520325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/902c01da-6632-47c1-80c1-c434756bdd6d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5mc2q\" (UID: \"902c01da-6632-47c1-80c1-c434756bdd6d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.520507 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.520486    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a0927c0-029c-459a-ada7-92c1ebc06c63-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-stqjh\" (UID: \"1a0927c0-029c-459a-ada7-92c1ebc06c63\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.690189 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.690157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"
Sep 17 17:37:41.740799 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.740769    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"
Sep 17 17:37:41.747510 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.747279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tz26h"
Sep 17 17:37:41.969541 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:41.969440    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh"]
Sep 17 17:37:41.971974 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:41.971947    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0927c0_029c_459a_ada7_92c1ebc06c63.slice/crio-8ad70e63f27e293d4246b794ce42ec4537ed2d1e95a4bef372cb96e3a96ba992 WatchSource:0}: Error finding container 8ad70e63f27e293d4246b794ce42ec4537ed2d1e95a4bef372cb96e3a96ba992: Status 404 returned error can't find the container with id 8ad70e63f27e293d4246b794ce42ec4537ed2d1e95a4bef372cb96e3a96ba992
Sep 17 17:37:42.027928 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.027837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-5mc2q"]
Sep 17 17:37:42.030425 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:42.030398    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902c01da_6632_47c1_80c1_c434756bdd6d.slice/crio-4b94da51b562291b24f42a677c2f74e3897ce6cccb08c6c8a97e75c7d8a9664f WatchSource:0}: Error finding container 4b94da51b562291b24f42a677c2f74e3897ce6cccb08c6c8a97e75c7d8a9664f: Status 404 returned error can't find the container with id 4b94da51b562291b24f42a677c2f74e3897ce6cccb08c6c8a97e75c7d8a9664f
Sep 17 17:37:42.458764 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.458721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz26h" event={"ID":"a0d72f17-d779-41b8-8450-0e6a76f53444","Type":"ContainerStarted","Data":"a868dd14e48f2b1cacedcba0feb13baaf7669fb95658fdfabcc95f3fe7007586"}
Sep 17 17:37:42.460086 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.460061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q" event={"ID":"902c01da-6632-47c1-80c1-c434756bdd6d","Type":"ContainerStarted","Data":"4b94da51b562291b24f42a677c2f74e3897ce6cccb08c6c8a97e75c7d8a9664f"}
Sep 17 17:37:42.462605 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.462579    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh" event={"ID":"1a0927c0-029c-459a-ada7-92c1ebc06c63","Type":"ContainerStarted","Data":"944fabdeba235facde1f9faedaf1a2b1ddcf2ffe8833c0de5840cdf4123c63f8"}
Sep 17 17:37:42.462714 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.462638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh" event={"ID":"1a0927c0-029c-459a-ada7-92c1ebc06c63","Type":"ContainerStarted","Data":"c4567f1501227e0128bf1e3f463b32a87064ab04697a5abfcce2bb106d30ff6e"}
Sep 17 17:37:42.462714 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.462669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh" event={"ID":"1a0927c0-029c-459a-ada7-92c1ebc06c63","Type":"ContainerStarted","Data":"8ad70e63f27e293d4246b794ce42ec4537ed2d1e95a4bef372cb96e3a96ba992"}
Sep 17 17:37:42.908116 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.908083    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-578db865db-6wv6l"]
Sep 17 17:37:42.912105 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.912084    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:42.915104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.914856    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eg2vfc3ulvlaa\""
Sep 17 17:37:42.915104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.914885    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:37:42.915104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.914940    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-whdks\""
Sep 17 17:37:42.915104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.914975    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:37:42.915104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.914856    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:37:42.915104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.914885    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:37:42.915450 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.915201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:37:42.924857 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:42.924838    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578db865db-6wv6l"]
Sep 17 17:37:43.032776 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.032734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.032788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrb6\" (UniqueName: \"kubernetes.io/projected/e175f2dd-3705-4b32-88ce-e9b6b3f014df-kube-api-access-mhrb6\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.032820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-grpc-tls\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.032884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e175f2dd-3705-4b32-88ce-e9b6b3f014df-metrics-client-ca\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.032916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.032986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.033015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-tls\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.033212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.033095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrb6\" (UniqueName: \"kubernetes.io/projected/e175f2dd-3705-4b32-88ce-e9b6b3f014df-kube-api-access-mhrb6\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-grpc-tls\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e175f2dd-3705-4b32-88ce-e9b6b3f014df-metrics-client-ca\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134487 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134487 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134634 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-tls\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.134984 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.134958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e175f2dd-3705-4b32-88ce-e9b6b3f014df-metrics-client-ca\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.137428 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.137409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-grpc-tls\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.137683 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.137656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.137768 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.137701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.137823 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.137761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.137957 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.137938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.138056 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.138037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e175f2dd-3705-4b32-88ce-e9b6b3f014df-secret-thanos-querier-tls\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.144464 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.144441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrb6\" (UniqueName: \"kubernetes.io/projected/e175f2dd-3705-4b32-88ce-e9b6b3f014df-kube-api-access-mhrb6\") pod \"thanos-querier-578db865db-6wv6l\" (UID: \"e175f2dd-3705-4b32-88ce-e9b6b3f014df\") " pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.222900 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.222865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:43.701867 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:43.701841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578db865db-6wv6l"]
Sep 17 17:37:43.703647 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:43.703608    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode175f2dd_3705_4b32_88ce_e9b6b3f014df.slice/crio-df105ed7968757d6f3d687682e0200f60b9be5af636259e43676ef6ae7172e95 WatchSource:0}: Error finding container df105ed7968757d6f3d687682e0200f60b9be5af636259e43676ef6ae7172e95: Status 404 returned error can't find the container with id df105ed7968757d6f3d687682e0200f60b9be5af636259e43676ef6ae7172e95
Sep 17 17:37:44.473453 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.473409    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q" event={"ID":"902c01da-6632-47c1-80c1-c434756bdd6d","Type":"ContainerStarted","Data":"0c974b4eac24664aadf94acb0d3b179df2cfcf1896bf25c56c32318876fdd4e4"}
Sep 17 17:37:44.473909 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.473478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q" event={"ID":"902c01da-6632-47c1-80c1-c434756bdd6d","Type":"ContainerStarted","Data":"d56061cfbfec5540b9947a9b18e3200808d29190ec2eedffa4c9425938da5f49"}
Sep 17 17:37:44.473909 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.473505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q" event={"ID":"902c01da-6632-47c1-80c1-c434756bdd6d","Type":"ContainerStarted","Data":"62ea05f26a3954efc54ed2faf71e69776ec06197c4483fad19f1a9d91c540b82"}
Sep 17 17:37:44.475969 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.475944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh" event={"ID":"1a0927c0-029c-459a-ada7-92c1ebc06c63","Type":"ContainerStarted","Data":"039daf997c6033f155e9b4b548371673f1e9ad4778c30d7325ec146ee5fc069c"}
Sep 17 17:37:44.477259 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.477239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"df105ed7968757d6f3d687682e0200f60b9be5af636259e43676ef6ae7172e95"}
Sep 17 17:37:44.478980 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.478952    2562 generic.go:356] "Generic (PLEG): container finished" podID="a0d72f17-d779-41b8-8450-0e6a76f53444" containerID="affac6e033e61f8c78fa6defcfe1370518bcff0d5beacfab9fd58f010d9bdbf8" exitCode=0
Sep 17 17:37:44.479080 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.478984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz26h" event={"ID":"a0d72f17-d779-41b8-8450-0e6a76f53444","Type":"ContainerDied","Data":"affac6e033e61f8c78fa6defcfe1370518bcff0d5beacfab9fd58f010d9bdbf8"}
Sep 17 17:37:44.504939 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.504864    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-5mc2q" podStartSLOduration=3.112074659 podStartE2EDuration="4.50484403s" totalImagesPullingTime="1.392769371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:44.503777658 +0000 UTC m=+208.951608139" watchObservedRunningTime="2026-09-17 17:37:44.50484403 +0000 UTC m=+208.952674487"
Sep 17 17:37:44.538076 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:44.537750    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-stqjh" podStartSLOduration=3.219412212 podStartE2EDuration="4.537728015s" totalImagesPullingTime="1.318315803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:44.535721865 +0000 UTC m=+208.983552342" watchObservedRunningTime="2026-09-17 17:37:44.537728015 +0000 UTC m=+208.985558463"
Sep 17 17:37:45.485541 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:45.485504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz26h" event={"ID":"a0d72f17-d779-41b8-8450-0e6a76f53444","Type":"ContainerStarted","Data":"76fd0c7e9cda443914508b6591cacfb12a3764f4af7bc94b77146356e2f7bd30"}
Sep 17 17:37:45.486132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:45.485595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tz26h" event={"ID":"a0d72f17-d779-41b8-8450-0e6a76f53444","Type":"ContainerStarted","Data":"4e0aec2882a9e1990d39b09dc2786b8be2ac3c2653c788273ecb99555c995b51"}
Sep 17 17:37:45.520407 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:45.520323    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tz26h" podStartSLOduration=3.784516685 podStartE2EDuration="5.520192938s" totalImagesPullingTime="1.650325418s" totalInitContainerRuntime="85.350835ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:45.517276778 +0000 UTC m=+209.965107285" watchObservedRunningTime="2026-09-17 17:37:45.520192938 +0000 UTC m=+209.968023396"
Sep 17 17:37:46.491669 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:46.491637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"d04d8222d4f31c118ca1714f5353a543872b033da2a5d3d4fe9565f452059d5f"}
Sep 17 17:37:46.492053 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:46.491703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"82c689312914d8630f23f07505f80994c1f0693d8550e1bff685617f78bd2522"}
Sep 17 17:37:46.492053 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:46.491729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"7eec9785a161fac67e716e03db8f0499726205309bab4556ea754b5cf062fd6c"}
Sep 17 17:37:47.391125 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.391092    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:47.394838 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.394821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.409402 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.409379    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:37:47.409402 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.409385    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:37:47.409750 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.409731    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:37:47.410802 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.410777    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:37:47.410953 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.410940    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:37:47.411001 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.410940    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:37:47.415416 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.415403    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:37:47.425277 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.425257    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:37:47.425350 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.425314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:37:47.425551 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.425538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5rejterrr199f\""
Sep 17 17:37:47.425605 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.425591    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:37:47.425676 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.425662    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:37:47.427466 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.427453    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-78tsv\""
Sep 17 17:37:47.427578 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.427562    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:37:47.431992 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.431973    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:37:47.433017 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.433002    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:37:47.482221 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.482197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:47.488349 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.488312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:37:47.498408 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.498382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"98cf82ed13868440f69f6c208990266ff8627f9be57d34228c55e7dd74fc5110"}
Sep 17 17:37:47.498693 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.498464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"2a2feda3b952de846469edb4906f5e170c7c7b0248f61c58f45dba197158a50b"}
Sep 17 17:37:47.498693 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.498500    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:47.498693 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.498527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" event={"ID":"e175f2dd-3705-4b32-88ce-e9b6b3f014df","Type":"ContainerStarted","Data":"dd9694e5aa98cd0682fdca97e08b3a1eb498e25a953e8d778a4806124ddd3f03"}
Sep 17 17:37:47.572821 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.572783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.572974 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.572849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.572974 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.572887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-config\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.572974 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.572916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.572976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6jl\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-kube-api-access-ml6jl\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.573556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.573511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.588611 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.588565    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l" podStartSLOduration=2.602068718 podStartE2EDuration="5.588548331s" totalImagesPullingTime="2.986479613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:47.585033617 +0000 UTC m=+212.032864078" watchObservedRunningTime="2026-09-17 17:37:47.588548331 +0000 UTC m=+212.036378786"
Sep 17 17:37:47.673935 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.673860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.673935 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.673896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.673935 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.673926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674184 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674239 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674363 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674430 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674482 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674482 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674582 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674582 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674677 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674677 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674763 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674763 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-config\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674763 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674752    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.674915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6jl\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-kube-api-access-ml6jl\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.675049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.674943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.676715 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.675524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.676715 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.676138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.677652 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.677082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.677652 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.677643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.678241 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.678214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.678777 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.678749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.678988 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.678961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.679139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.679119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.679244 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.679222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.679650 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.679608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.679733 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.679668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.679849 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.679812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.679932 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.679902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.680600 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.680582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.680945 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.680923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-config\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.681446 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.681422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.682422 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.682402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.682624 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.682607    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.691301 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.691283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6jl\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-kube-api-access-ml6jl\") pod \"prometheus-k8s-0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.704054 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.704035    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:47.991747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:47.991726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:47.993275 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:37:47.993253    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae73aa47_5788_46c7_abf0_dea8434d49f0.slice/crio-0fa82f07f2e20348d5e6edfe37b376768bce6bccae16309e332e7404b17a14dc WatchSource:0}: Error finding container 0fa82f07f2e20348d5e6edfe37b376768bce6bccae16309e332e7404b17a14dc: Status 404 returned error can't find the container with id 0fa82f07f2e20348d5e6edfe37b376768bce6bccae16309e332e7404b17a14dc
Sep 17 17:37:48.503343 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:48.503289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"0fa82f07f2e20348d5e6edfe37b376768bce6bccae16309e332e7404b17a14dc"}
Sep 17 17:37:49.508501 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:49.508412    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="5376a562c0a4f2e263ac7bb1f93dbf9aba9051de9f7e72779f87868dfc950f67" exitCode=0
Sep 17 17:37:49.508821 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:49.508510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"5376a562c0a4f2e263ac7bb1f93dbf9aba9051de9f7e72779f87868dfc950f67"}
Sep 17 17:37:50.721815 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:50.721782    2562 scope.go:117] "RemoveContainer" containerID="1b5067628be3abfdc4719ef724002348868fa9f1505d6b38ca98f52416eed23d"
Sep 17 17:37:51.519960 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:51.519929    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:37:51.520138 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:51.519996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" event={"ID":"7de942b3-1609-4be9-9632-b39a80a5d381","Type":"ContainerStarted","Data":"7f560093b14224a48606ce4067056c87157f49cedb1f98414b28d8455535aa0c"}
Sep 17 17:37:51.520303 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:51.520283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:37:51.543010 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:51.542896    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9" podStartSLOduration=51.266356292 podStartE2EDuration="56.542877197s" totalImagesPullingTime="5.276520905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:51.541991491 +0000 UTC m=+215.989821972" watchObservedRunningTime="2026-09-17 17:37:51.542877197 +0000 UTC m=+215.990707654"
Sep 17 17:37:51.938318 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:51.938141    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-fn7x9"
Sep 17 17:37:52.448188 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.448121    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"]
Sep 17 17:37:52.448491 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:37:52.448457    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw" podUID="82a1fb15-93eb-46b6-a334-5ca33f438d14"
Sep 17 17:37:52.523993 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.523959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:37:52.532598 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.532568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:37:52.628082 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628053    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-certificates\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628245 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628095    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs26z\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-kube-api-access-zs26z\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628245 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628134    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-image-registry-private-configuration\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628245 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-bound-sa-token\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628245 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82a1fb15-93eb-46b6-a334-5ca33f438d14-ca-trust-extracted\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628473 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628271    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-installation-pull-secrets\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628473 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-trusted-ca\") pod \"82a1fb15-93eb-46b6-a334-5ca33f438d14\" (UID: \"82a1fb15-93eb-46b6-a334-5ca33f438d14\") "
Sep 17 17:37:52.628574 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628524    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:52.628574 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a1fb15-93eb-46b6-a334-5ca33f438d14-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:52.628705 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628683    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-certificates\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:52.628824 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628717    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82a1fb15-93eb-46b6-a334-5ca33f438d14-ca-trust-extracted\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:52.628898 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.628881    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:52.630637 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.630616    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-kube-api-access-zs26z" (OuterVolumeSpecName: "kube-api-access-zs26z") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "kube-api-access-zs26z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:52.630755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.630615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:52.630905 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.630886    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:52.630980 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.630958    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "82a1fb15-93eb-46b6-a334-5ca33f438d14" (UID: "82a1fb15-93eb-46b6-a334-5ca33f438d14"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:52.730047 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.730028    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a1fb15-93eb-46b6-a334-5ca33f438d14-trusted-ca\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:52.730127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.730051    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs26z\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-kube-api-access-zs26z\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:52.730127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.730060    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-image-registry-private-configuration\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:52.730127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.730074    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-bound-sa-token\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:52.730127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:52.730086    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82a1fb15-93eb-46b6-a334-5ca33f438d14-installation-pull-secrets\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:53.510767 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.510736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-578db865db-6wv6l"
Sep 17 17:37:53.531594 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"
Sep 17 17:37:53.531594 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"ab99e1fd7167808f33107198a8d820277539d38210fea500d456948e45964ffb"}
Sep 17 17:37:53.531796 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"07a3ce62bc00b11add73f310787e7d33ac33309f8d2e30821c02f5db47eb1f05"}
Sep 17 17:37:53.531796 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"29e118c813b9cea0c99d3a7d5783d41957e899ce0f18dd8d7bb0775c7c176e9b"}
Sep 17 17:37:53.531796 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531699    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"29e9158f5f9569c6439f1bfe1e29664d55b4fbaae88a53011faf00a822c7ac69"}
Sep 17 17:37:53.531796 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"239bf7f41e51df21f1870339337bdff4c07ed818244c213719c584a00dd3c0e5"}
Sep 17 17:37:53.531796 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.531754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerStarted","Data":"9ca00d6ab18e753217c97ba0fa0bf8b7bb9159bce0da08d88beb4ffb33ba7895"}
Sep 17 17:37:53.590237 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.590204    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"]
Sep 17 17:37:53.594197 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.594169    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d74d8ff8d-fxlfw"]
Sep 17 17:37:53.631625 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.631574    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3434389429999998 podStartE2EDuration="6.631556181s" totalImagesPullingTime="4.269894746s" totalInitContainerRuntime="18.222492ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:53.630621445 +0000 UTC m=+218.078451932" watchObservedRunningTime="2026-09-17 17:37:53.631556181 +0000 UTC m=+218.079386697"
Sep 17 17:37:53.639458 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.639423    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82a1fb15-93eb-46b6-a334-5ca33f438d14-registry-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:37:53.726974 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:53.726941    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a1fb15-93eb-46b6-a334-5ca33f438d14" path="/var/lib/kubelet/pods/82a1fb15-93eb-46b6-a334-5ca33f438d14/volumes"
Sep 17 17:37:57.704768 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:37:57.704727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:24.647948 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:24.647913    2562 generic.go:356] "Generic (PLEG): container finished" podID="8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13" containerID="43359f29952efafb93b3f097334087d09cf0a68c352bee500343da3afad9c68f" exitCode=0
Sep 17 17:38:24.648383 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:24.647986    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr" event={"ID":"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13","Type":"ContainerDied","Data":"43359f29952efafb93b3f097334087d09cf0a68c352bee500343da3afad9c68f"}
Sep 17 17:38:24.648383 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:24.648282    2562 scope.go:117] "RemoveContainer" containerID="43359f29952efafb93b3f097334087d09cf0a68c352bee500343da3afad9c68f"
Sep 17 17:38:25.653737 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:25.653697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-bjwcr" event={"ID":"8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13","Type":"ContainerStarted","Data":"c6b99778b6f66baf03bc11dd569ddc07a5cc0cc43fd818d7485a78bb4f5a20f1"}
Sep 17 17:38:29.455006 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:29.454914    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:38:29.457306 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:29.457279    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc60926-841c-4824-872b-30a7c806ade0-metrics-certs\") pod \"network-metrics-daemon-rzpqd\" (UID: \"fbc60926-841c-4824-872b-30a7c806ade0\") " pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:38:29.625838 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:29.625809    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g24m8\""
Sep 17 17:38:29.633098 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:29.633084    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rzpqd"
Sep 17 17:38:29.904751 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:29.904622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rzpqd"]
Sep 17 17:38:29.907635 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:38:29.907602    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc60926_841c_4824_872b_30a7c806ade0.slice/crio-90bbf3dbd78034a2d035564309dc702d5d234b0fe92695877335441f87e7b70f WatchSource:0}: Error finding container 90bbf3dbd78034a2d035564309dc702d5d234b0fe92695877335441f87e7b70f: Status 404 returned error can't find the container with id 90bbf3dbd78034a2d035564309dc702d5d234b0fe92695877335441f87e7b70f
Sep 17 17:38:30.674079 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:30.674042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzpqd" event={"ID":"fbc60926-841c-4824-872b-30a7c806ade0","Type":"ContainerStarted","Data":"90bbf3dbd78034a2d035564309dc702d5d234b0fe92695877335441f87e7b70f"}
Sep 17 17:38:31.679823 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:31.679788    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzpqd" event={"ID":"fbc60926-841c-4824-872b-30a7c806ade0","Type":"ContainerStarted","Data":"eb340b20a2dce4c7ca446d3514d2d5efe164e8c2b30904071fe0ff73b7060017"}
Sep 17 17:38:31.680180 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:31.679850    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rzpqd" event={"ID":"fbc60926-841c-4824-872b-30a7c806ade0","Type":"ContainerStarted","Data":"203ce4d396d35c24686a1c822dd5b5732aa14f8c1a427e92376783252c8f98ec"}
Sep 17 17:38:31.702167 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:31.702101    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rzpqd" podStartSLOduration=252.829795571 podStartE2EDuration="4m13.702082362s" totalImagesPullingTime="872.286791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:31.701537806 +0000 UTC m=+256.149368287" watchObservedRunningTime="2026-09-17 17:38:31.702082362 +0000 UTC m=+256.149912818"
Sep 17 17:38:31.799225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:31.799197    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mwgsw_6ebdeed2-06ee-41de-b637-0ae050cf8c63/dns-node-resolver/0.log"
Sep 17 17:38:47.704404 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:47.704364    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:47.727385 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:47.727305    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:47.755320 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:47.755296    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:57.278455 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:38:57.278397    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bljxr" podUID="c86541f5-fa09-4a99-b838-01322d9c0cb6"
Sep 17 17:38:57.278455 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:38:57.278397    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp" podUID="d916b8ba-958f-4cb4-aadc-ab6f3467ac17"
Sep 17 17:38:57.782902 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:57.782873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bljxr"
Sep 17 17:38:57.782902 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:38:57.782905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:39:01.019045 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.019009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:39:01.021285 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.021265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d916b8ba-958f-4cb4-aadc-ab6f3467ac17-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8lqdp\" (UID: \"d916b8ba-958f-4cb4-aadc-ab6f3467ac17\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:39:01.086009 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.085982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-95zq5\""
Sep 17 17:39:01.094450 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.094432    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"
Sep 17 17:39:01.120365 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.120325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:39:01.122532 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.122504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c86541f5-fa09-4a99-b838-01322d9c0cb6-metrics-tls\") pod \"dns-default-bljxr\" (UID: \"c86541f5-fa09-4a99-b838-01322d9c0cb6\") " pod="openshift-dns/dns-default-bljxr"
Sep 17 17:39:01.386136 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.386112    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqqq4\""
Sep 17 17:39:01.393612 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.393579    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bljxr"
Sep 17 17:39:01.394661 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.394617    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp"]
Sep 17 17:39:01.397650 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:39:01.397629    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd916b8ba_958f_4cb4_aadc_ab6f3467ac17.slice/crio-f4f90d4e4a84923896f7e8ad3b5c408cccc210503a972f891f517166f9b8adaf WatchSource:0}: Error finding container f4f90d4e4a84923896f7e8ad3b5c408cccc210503a972f891f517166f9b8adaf: Status 404 returned error can't find the container with id f4f90d4e4a84923896f7e8ad3b5c408cccc210503a972f891f517166f9b8adaf
Sep 17 17:39:01.663882 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.663860    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bljxr"]
Sep 17 17:39:01.665613 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:39:01.665593    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86541f5_fa09_4a99_b838_01322d9c0cb6.slice/crio-4819e66758ca74d39592411cc4d8189372f5cece8f8abd358bbeab985f25e6c4 WatchSource:0}: Error finding container 4819e66758ca74d39592411cc4d8189372f5cece8f8abd358bbeab985f25e6c4: Status 404 returned error can't find the container with id 4819e66758ca74d39592411cc4d8189372f5cece8f8abd358bbeab985f25e6c4
Sep 17 17:39:01.799430 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.799399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bljxr" event={"ID":"c86541f5-fa09-4a99-b838-01322d9c0cb6","Type":"ContainerStarted","Data":"4819e66758ca74d39592411cc4d8189372f5cece8f8abd358bbeab985f25e6c4"}
Sep 17 17:39:01.800574 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:01.800553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp" event={"ID":"d916b8ba-958f-4cb4-aadc-ab6f3467ac17","Type":"ContainerStarted","Data":"f4f90d4e4a84923896f7e8ad3b5c408cccc210503a972f891f517166f9b8adaf"}
Sep 17 17:39:02.805917 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:02.805886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp" event={"ID":"d916b8ba-958f-4cb4-aadc-ab6f3467ac17","Type":"ContainerStarted","Data":"fd782e052f16e0959c8e833f4f3016f8ce37823b2060af94a4f6ec85ca3e53de"}
Sep 17 17:39:02.833781 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:02.833720    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8lqdp" podStartSLOduration=273.570423939 podStartE2EDuration="4m34.833700508s" totalImagesPullingTime="1.263276569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:02.832507621 +0000 UTC m=+287.280338103" watchObservedRunningTime="2026-09-17 17:39:02.833700508 +0000 UTC m=+287.281530996"
Sep 17 17:39:03.811274 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:03.811232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bljxr" event={"ID":"c86541f5-fa09-4a99-b838-01322d9c0cb6","Type":"ContainerStarted","Data":"322112eb120c56efe41eb7adb45edca84dcd22a9edf7defc19fd306a51dcbd9d"}
Sep 17 17:39:03.811671 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:03.811296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bljxr" event={"ID":"c86541f5-fa09-4a99-b838-01322d9c0cb6","Type":"ContainerStarted","Data":"41d62367f95d9a9256da05d639de47a063ce285ae433b017a568080258d4bdaa"}
Sep 17 17:39:03.811671 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:03.811325    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bljxr"
Sep 17 17:39:03.831389 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:03.831342    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bljxr" podStartSLOduration=251.095597811 podStartE2EDuration="4m12.831311461s" totalImagesPullingTime="1.73571365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:03.830244182 +0000 UTC m=+288.278074681" watchObservedRunningTime="2026-09-17 17:39:03.831311461 +0000 UTC m=+288.279141916"
Sep 17 17:39:05.615762 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.615728    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:39:05.616628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.616481    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy" containerID="cri-o://07a3ce62bc00b11add73f310787e7d33ac33309f8d2e30821c02f5db47eb1f05" gracePeriod=600
Sep 17 17:39:05.616628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.616508    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="config-reloader" containerID="cri-o://239bf7f41e51df21f1870339337bdff4c07ed818244c213719c584a00dd3c0e5" gracePeriod=600
Sep 17 17:39:05.616628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.616508    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="thanos-sidecar" containerID="cri-o://29e9158f5f9569c6439f1bfe1e29664d55b4fbaae88a53011faf00a822c7ac69" gracePeriod=600
Sep 17 17:39:05.616628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.616535    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ab99e1fd7167808f33107198a8d820277539d38210fea500d456948e45964ffb" gracePeriod=600
Sep 17 17:39:05.616628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.616481    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="prometheus" containerID="cri-o://9ca00d6ab18e753217c97ba0fa0bf8b7bb9159bce0da08d88beb4ffb33ba7895" gracePeriod=600
Sep 17 17:39:05.616628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.616535    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-web" containerID="cri-o://29e118c813b9cea0c99d3a7d5783d41957e899ce0f18dd8d7bb0775c7c176e9b" gracePeriod=600
Sep 17 17:39:05.824168 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824138    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="ab99e1fd7167808f33107198a8d820277539d38210fea500d456948e45964ffb" exitCode=0
Sep 17 17:39:05.824168 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824162    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="07a3ce62bc00b11add73f310787e7d33ac33309f8d2e30821c02f5db47eb1f05" exitCode=0
Sep 17 17:39:05.824168 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824169    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="29e9158f5f9569c6439f1bfe1e29664d55b4fbaae88a53011faf00a822c7ac69" exitCode=0
Sep 17 17:39:05.824168 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824174    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="239bf7f41e51df21f1870339337bdff4c07ed818244c213719c584a00dd3c0e5" exitCode=0
Sep 17 17:39:05.824168 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824179    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="9ca00d6ab18e753217c97ba0fa0bf8b7bb9159bce0da08d88beb4ffb33ba7895" exitCode=0
Sep 17 17:39:05.824557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"ab99e1fd7167808f33107198a8d820277539d38210fea500d456948e45964ffb"}
Sep 17 17:39:05.824557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"07a3ce62bc00b11add73f310787e7d33ac33309f8d2e30821c02f5db47eb1f05"}
Sep 17 17:39:05.824557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"29e9158f5f9569c6439f1bfe1e29664d55b4fbaae88a53011faf00a822c7ac69"}
Sep 17 17:39:05.824557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"239bf7f41e51df21f1870339337bdff4c07ed818244c213719c584a00dd3c0e5"}
Sep 17 17:39:05.824557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:05.824443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"9ca00d6ab18e753217c97ba0fa0bf8b7bb9159bce0da08d88beb4ffb33ba7895"}
Sep 17 17:39:06.833038 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.833011    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerID="29e118c813b9cea0c99d3a7d5783d41957e899ce0f18dd8d7bb0775c7c176e9b" exitCode=0
Sep 17 17:39:06.833379 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.833082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"29e118c813b9cea0c99d3a7d5783d41957e899ce0f18dd8d7bb0775c7c176e9b"}
Sep 17 17:39:06.854204 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.854179    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:06.970618 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970546    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-trusted-ca-bundle\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970618 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-tls-assets\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970618 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970598    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-db\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970618 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970616    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-tls\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-2\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970662    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970687    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-config\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-web-config\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-serving-certs-ca-bundle\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970789    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-0\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.970906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.970824    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-metrics-client-ca\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.971235 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971069    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.971235 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.971682 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971642    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.971903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-config-out\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-kubelet-serving-ca-bundle\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-kube-rbac-proxy\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971909    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972155    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml6jl\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-kube-api-access-ml6jl\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-grpc-tls\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-metrics-client-certs\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972283    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-thanos-prometheus-http-client-file\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972395    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-1\") pod \"ae73aa47-5788-46c7-abf0-dea8434d49f0\" (UID: \"ae73aa47-5788-46c7-abf0-dea8434d49f0\") "
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972642    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972658    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972672    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972688    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-metrics-client-ca\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.971993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972481    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.975399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.972967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:06.976305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.974170    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:06.976305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.974755    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.976305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.975196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-config-out" (OuterVolumeSpecName: "config-out") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:06.976305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.975527    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-kube-api-access-ml6jl" (OuterVolumeSpecName: "kube-api-access-ml6jl") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "kube-api-access-ml6jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:06.976305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.975869    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.976841 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.976808    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.977012 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.976945    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.977091 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.977068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-config" (OuterVolumeSpecName: "config") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.977091 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.977079    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.977351 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.977310    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.978110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.978092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:06.986712 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:06.986691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-web-config" (OuterVolumeSpecName: "web-config") pod "ae73aa47-5788-46c7-abf0-dea8434d49f0" (UID: "ae73aa47-5788-46c7-abf0-dea8434d49f0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:07.073753 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073716    2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-kube-rbac-proxy\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073753 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073753    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073753 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073764    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6jl\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-kube-api-access-ml6jl\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073773    2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-grpc-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073781    2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-metrics-client-certs\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073789    2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073798    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073806    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae73aa47-5788-46c7-abf0-dea8434d49f0-tls-assets\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073814    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-prometheus-k8s-db\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073822    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073831    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073840    2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073847    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae73aa47-5788-46c7-abf0-dea8434d49f0-web-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073854    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073862    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae73aa47-5788-46c7-abf0-dea8434d49f0-config-out\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.073903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.073870    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae73aa47-5788-46c7-abf0-dea8434d49f0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.840267 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.840235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ae73aa47-5788-46c7-abf0-dea8434d49f0","Type":"ContainerDied","Data":"0fa82f07f2e20348d5e6edfe37b376768bce6bccae16309e332e7404b17a14dc"}
Sep 17 17:39:07.840267 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.840271    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:07.840669 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.840348    2562 scope.go:117] "RemoveContainer" containerID="ab99e1fd7167808f33107198a8d820277539d38210fea500d456948e45964ffb"
Sep 17 17:39:07.848903 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.848883    2562 scope.go:117] "RemoveContainer" containerID="07a3ce62bc00b11add73f310787e7d33ac33309f8d2e30821c02f5db47eb1f05"
Sep 17 17:39:07.857519 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.857497    2562 scope.go:117] "RemoveContainer" containerID="29e118c813b9cea0c99d3a7d5783d41957e899ce0f18dd8d7bb0775c7c176e9b"
Sep 17 17:39:07.864815 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.864359    2562 scope.go:117] "RemoveContainer" containerID="29e9158f5f9569c6439f1bfe1e29664d55b4fbaae88a53011faf00a822c7ac69"
Sep 17 17:39:07.866361 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.866342    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:39:07.871151 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.871131    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:39:07.872632 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.872615    2562 scope.go:117] "RemoveContainer" containerID="239bf7f41e51df21f1870339337bdff4c07ed818244c213719c584a00dd3c0e5"
Sep 17 17:39:07.879369 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.879351    2562 scope.go:117] "RemoveContainer" containerID="9ca00d6ab18e753217c97ba0fa0bf8b7bb9159bce0da08d88beb4ffb33ba7895"
Sep 17 17:39:07.886180 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.886164    2562 scope.go:117] "RemoveContainer" containerID="5376a562c0a4f2e263ac7bb1f93dbf9aba9051de9f7e72779f87868dfc950f67"
Sep 17 17:39:07.901907 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.901887    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:39:07.902174 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902163    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="config-reloader"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902175    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="config-reloader"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902187    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="thanos-sidecar"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902192    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="thanos-sidecar"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="prometheus"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902206    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="prometheus"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902214    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="init-config-reloader"
Sep 17 17:39:07.902219 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902219    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="init-config-reloader"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902227    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902233    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902244    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-web"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902250    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-web"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902258    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-thanos"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902263    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-thanos"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902309    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-web"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902317    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902323    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="kube-rbac-proxy-thanos"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902342    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="config-reloader"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902350    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="prometheus"
Sep 17 17:39:07.902443 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.902356    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" containerName="thanos-sidecar"
Sep 17 17:39:07.905977 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.905962    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:07.908907 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.908854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:39:07.908907 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.908867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:39:07.909086 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909072    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5rejterrr199f\""
Sep 17 17:39:07.909183 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909164    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:39:07.909299 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:39:07.909299 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909214    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:39:07.909966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909882    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:39:07.909966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909911    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:39:07.910095 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.909995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:39:07.911011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.910969    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:39:07.911110 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.911098    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:39:07.911289 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.911245    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:39:07.911289 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.911281    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:39:07.912479 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.912462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-78tsv\""
Sep 17 17:39:07.913372 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.913354    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:39:07.914628 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.914606    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:39:07.918513 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.918495    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:39:07.929200 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:07.929179    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:39:08.080019 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.079977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-config\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ft7d\" (UniqueName: \"kubernetes.io/projected/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-kube-api-access-9ft7d\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080423    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080496    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080846 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080846 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.080846 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.080590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181177 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181118    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-config\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181177 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181441 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ft7d\" (UniqueName: \"kubernetes.io/projected/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-kube-api-access-9ft7d\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181441 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181441 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181441 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.181976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.181877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.182374 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.182245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.182374 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.182355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.182487 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.182424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.184027 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.182757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.184684 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.184452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-config\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.184684 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.184545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.184684 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.184590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.184957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185104 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.185060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185224 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.185200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185360 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.185316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185440 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.185399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185657 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.185631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.185913 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.185893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.186657 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.186634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.187552 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.187533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.187629 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.187587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.187629 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.187616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.187708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.187633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.190349 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.190309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ft7d\" (UniqueName: \"kubernetes.io/projected/64119c2b-c0e1-4d1c-b676-0518d0a1af4f-kube-api-access-9ft7d\") pod \"prometheus-k8s-0\" (UID: \"64119c2b-c0e1-4d1c-b676-0518d0a1af4f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.216442 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.216414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:08.503466 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.503443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:39:08.505470 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:39:08.505434    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64119c2b_c0e1_4d1c_b676_0518d0a1af4f.slice/crio-31be0dfeeebaaa96a32706b13d5df166478fb03deaaa537426ba364d0c2c1fd3 WatchSource:0}: Error finding container 31be0dfeeebaaa96a32706b13d5df166478fb03deaaa537426ba364d0c2c1fd3: Status 404 returned error can't find the container with id 31be0dfeeebaaa96a32706b13d5df166478fb03deaaa537426ba364d0c2c1fd3
Sep 17 17:39:08.847368 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.847321    2562 generic.go:356] "Generic (PLEG): container finished" podID="64119c2b-c0e1-4d1c-b676-0518d0a1af4f" containerID="bd666bb051f3a5701b3f0cb6e5257b067687a5901c4c80ebb37ddabe48c08df1" exitCode=0
Sep 17 17:39:08.847720 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.847370    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerDied","Data":"bd666bb051f3a5701b3f0cb6e5257b067687a5901c4c80ebb37ddabe48c08df1"}
Sep 17 17:39:08.847720 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:08.847439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"31be0dfeeebaaa96a32706b13d5df166478fb03deaaa537426ba364d0c2c1fd3"}
Sep 17 17:39:09.728224 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.728184    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae73aa47-5788-46c7-abf0-dea8434d49f0" path="/var/lib/kubelet/pods/ae73aa47-5788-46c7-abf0-dea8434d49f0/volumes"
Sep 17 17:39:09.855556 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.855522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"aabd6d2180107697e32d11b6d906b08b0949eb189fca9486a52ecc722fc183f7"}
Sep 17 17:39:09.855978 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.855586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"be0423aaa1585de31d9937b719ff4501fb2740b8fcd509863f49ad95934b3713"}
Sep 17 17:39:09.855978 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.855611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"67f233ac004821587d5ae60b63a14486bf9ee5509def573a66301773127f05b5"}
Sep 17 17:39:09.855978 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.855650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"ff7cb7f1485a8c55db160ac01bfa0eb61726a38c50a258460182c06b307fd54b"}
Sep 17 17:39:09.855978 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.855679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"a5950faa55737af410a21974aaf1c23514a66a58f99f7f5c2d0a2afa2b748527"}
Sep 17 17:39:09.855978 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.855701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"64119c2b-c0e1-4d1c-b676-0518d0a1af4f","Type":"ContainerStarted","Data":"909d8bef740a29d871d4eeb3118997bd51666df858190985156b3f32e5b92175"}
Sep 17 17:39:09.893658 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:09.893605    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8731201410000002 podStartE2EDuration="2.893589011s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.46887ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:09.892609625 +0000 UTC m=+294.340440081" watchObservedRunningTime="2026-09-17 17:39:09.893589011 +0000 UTC m=+294.341419466"
Sep 17 17:39:13.217546 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:13.217502    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:13.816734 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:13.816704    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bljxr"
Sep 17 17:39:17.625132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:17.625105    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:39:17.632157 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:17.632134    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:39:17.636995 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:39:17.636965    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:40:08.216704 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:40:08.216617    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:40:08.232306 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:40:08.232280    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:40:09.112602 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:40:09.112576    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:43:07.798864 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.798780    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-4ncs4"]
Sep 17 17:43:07.802493 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.802472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:07.806844 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.806788    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:43:07.806844 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.806809    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:43:07.806844 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.806817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kcx4l\""
Sep 17 17:43:07.807771 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.807752    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:43:07.829416 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.829383    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1e104153-ae7f-42f9-a8b7-7401a1ee2a4f-data\") pod \"seaweedfs-64568bcd49-4ncs4\" (UID: \"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f\") " pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:07.829416 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.829411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqc2\" (UniqueName: \"kubernetes.io/projected/1e104153-ae7f-42f9-a8b7-7401a1ee2a4f-kube-api-access-dxqc2\") pod \"seaweedfs-64568bcd49-4ncs4\" (UID: \"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f\") " pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:07.836596 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.836571    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4ncs4"]
Sep 17 17:43:07.930458 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.930418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1e104153-ae7f-42f9-a8b7-7401a1ee2a4f-data\") pod \"seaweedfs-64568bcd49-4ncs4\" (UID: \"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f\") " pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:07.930637 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.930464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqc2\" (UniqueName: \"kubernetes.io/projected/1e104153-ae7f-42f9-a8b7-7401a1ee2a4f-kube-api-access-dxqc2\") pod \"seaweedfs-64568bcd49-4ncs4\" (UID: \"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f\") " pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:07.930790 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.930770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1e104153-ae7f-42f9-a8b7-7401a1ee2a4f-data\") pod \"seaweedfs-64568bcd49-4ncs4\" (UID: \"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f\") " pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:07.939954 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:07.939921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqc2\" (UniqueName: \"kubernetes.io/projected/1e104153-ae7f-42f9-a8b7-7401a1ee2a4f-kube-api-access-dxqc2\") pod \"seaweedfs-64568bcd49-4ncs4\" (UID: \"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f\") " pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:08.113623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:08.113582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:08.389226 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:08.389201    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4ncs4"]
Sep 17 17:43:08.391692 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:43:08.391670    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e104153_ae7f_42f9_a8b7_7401a1ee2a4f.slice/crio-1ae163ecc08c19e5550d43c5d49ac4282b2b220f8f457630d55fb7d65ad9100b WatchSource:0}: Error finding container 1ae163ecc08c19e5550d43c5d49ac4282b2b220f8f457630d55fb7d65ad9100b: Status 404 returned error can't find the container with id 1ae163ecc08c19e5550d43c5d49ac4282b2b220f8f457630d55fb7d65ad9100b
Sep 17 17:43:08.393130 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:08.393098    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:43:08.819290 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:08.819261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4ncs4" event={"ID":"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f","Type":"ContainerStarted","Data":"1ae163ecc08c19e5550d43c5d49ac4282b2b220f8f457630d55fb7d65ad9100b"}
Sep 17 17:43:11.833889 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:11.833857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4ncs4" event={"ID":"1e104153-ae7f-42f9-a8b7-7401a1ee2a4f","Type":"ContainerStarted","Data":"5d2544ef3c02417a2244e53be2c81db18f9aff45ce88da17d880d20305f5a70c"}
Sep 17 17:43:11.834270 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:11.833926    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:43:11.879558 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:11.879509    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-4ncs4" podStartSLOduration=1.739073319 podStartE2EDuration="4.879494093s" totalImagesPullingTime="3.140420774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:11.878408168 +0000 UTC m=+536.326238624" watchObservedRunningTime="2026-09-17 17:43:11.879494093 +0000 UTC m=+536.327324546"
Sep 17 17:43:17.840935 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:43:17.840903    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-4ncs4"
Sep 17 17:44:17.653906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:17.653869    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:44:17.659632 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:17.659610    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:44:41.142681 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.142612    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jgrrc"]
Sep 17 17:44:41.146048 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.146028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.149143 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.149125    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:44:41.156673 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.156651    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jgrrc"]
Sep 17 17:44:41.294889 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.294854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwwm\" (UniqueName: \"kubernetes.io/projected/db08db2e-17e4-426c-89d4-ae8ab7d03b89-kube-api-access-jtwwm\") pod \"s3-init-jgrrc\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") " pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.294889 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.294895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db08db2e-17e4-426c-89d4-ae8ab7d03b89-model-cache\") pod \"s3-init-jgrrc\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") " pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.395227 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.395173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwwm\" (UniqueName: \"kubernetes.io/projected/db08db2e-17e4-426c-89d4-ae8ab7d03b89-kube-api-access-jtwwm\") pod \"s3-init-jgrrc\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") " pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.395227 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.395203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db08db2e-17e4-426c-89d4-ae8ab7d03b89-model-cache\") pod \"s3-init-jgrrc\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") " pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.395531 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.395517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db08db2e-17e4-426c-89d4-ae8ab7d03b89-model-cache\") pod \"s3-init-jgrrc\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") " pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.405453 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.405432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwwm\" (UniqueName: \"kubernetes.io/projected/db08db2e-17e4-426c-89d4-ae8ab7d03b89-kube-api-access-jtwwm\") pod \"s3-init-jgrrc\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") " pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.471161 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.471141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jgrrc"
Sep 17 17:44:41.741607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:41.741573    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jgrrc"]
Sep 17 17:44:41.744261 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:44:41.744228    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb08db2e_17e4_426c_89d4_ae8ab7d03b89.slice/crio-aef10bb614523922aea9ef2ba181793a99fa3003af0adbec31ef6bd320750248 WatchSource:0}: Error finding container aef10bb614523922aea9ef2ba181793a99fa3003af0adbec31ef6bd320750248: Status 404 returned error can't find the container with id aef10bb614523922aea9ef2ba181793a99fa3003af0adbec31ef6bd320750248
Sep 17 17:44:42.196453 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:42.196423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jgrrc" event={"ID":"db08db2e-17e4-426c-89d4-ae8ab7d03b89","Type":"ContainerStarted","Data":"aef10bb614523922aea9ef2ba181793a99fa3003af0adbec31ef6bd320750248"}
Sep 17 17:44:46.215955 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:46.215925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jgrrc" event={"ID":"db08db2e-17e4-426c-89d4-ae8ab7d03b89","Type":"ContainerStarted","Data":"495e9f4fa7f813ac7c6772b5b069e8585ce8e2ddd2e3301e49ab34a216abfb05"}
Sep 17 17:44:48.226131 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:48.226089    2562 generic.go:356] "Generic (PLEG): container finished" podID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerID="495e9f4fa7f813ac7c6772b5b069e8585ce8e2ddd2e3301e49ab34a216abfb05" exitCode=0
Sep 17 17:44:48.226533 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:48.226157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jgrrc" event={"ID":"db08db2e-17e4-426c-89d4-ae8ab7d03b89","Type":"ContainerDied","Data":"495e9f4fa7f813ac7c6772b5b069e8585ce8e2ddd2e3301e49ab34a216abfb05"}
Sep 17 17:44:53.253697 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:53.253661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jgrrc" event={"ID":"db08db2e-17e4-426c-89d4-ae8ab7d03b89","Type":"ContainerStarted","Data":"fa49722929a4e5ffa7f23a1fb0656e1bbf453bcdbc77655861ae0501ad2c492f"}
Sep 17 17:44:53.276667 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:44:53.276607    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-jgrrc" podStartSLOduration=2.03497511 podStartE2EDuration="12.276588412s" totalImagesPullingTime="8.40845029s" totalInitContainerRuntime="1.833163012s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:53.27532446 +0000 UTC m=+637.723154916" watchObservedRunningTime="2026-09-17 17:44:53.276588412 +0000 UTC m=+637.724418871"
Sep 17 17:45:57.524835 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:57.524807    2562 generic.go:356] "Generic (PLEG): container finished" podID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerID="fa49722929a4e5ffa7f23a1fb0656e1bbf453bcdbc77655861ae0501ad2c492f" exitCode=0
Sep 17 17:45:57.525156 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:57.524845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jgrrc" event={"ID":"db08db2e-17e4-426c-89d4-ae8ab7d03b89","Type":"ContainerDied","Data":"fa49722929a4e5ffa7f23a1fb0656e1bbf453bcdbc77655861ae0501ad2c492f"}
Sep 17 17:45:58.645827 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.645805    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jgrrc"
Sep 17 17:45:58.797886 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.797804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtwwm\" (UniqueName: \"kubernetes.io/projected/db08db2e-17e4-426c-89d4-ae8ab7d03b89-kube-api-access-jtwwm\") pod \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") "
Sep 17 17:45:58.797886 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.797883    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db08db2e-17e4-426c-89d4-ae8ab7d03b89-model-cache\") pod \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\" (UID: \"db08db2e-17e4-426c-89d4-ae8ab7d03b89\") "
Sep 17 17:45:58.800048 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.800012    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db08db2e-17e4-426c-89d4-ae8ab7d03b89-kube-api-access-jtwwm" (OuterVolumeSpecName: "kube-api-access-jtwwm") pod "db08db2e-17e4-426c-89d4-ae8ab7d03b89" (UID: "db08db2e-17e4-426c-89d4-ae8ab7d03b89"). InnerVolumeSpecName "kube-api-access-jtwwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:58.816311 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.816284    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db08db2e-17e4-426c-89d4-ae8ab7d03b89-model-cache" (OuterVolumeSpecName: "model-cache") pod "db08db2e-17e4-426c-89d4-ae8ab7d03b89" (UID: "db08db2e-17e4-426c-89d4-ae8ab7d03b89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:58.898985 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.898961    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db08db2e-17e4-426c-89d4-ae8ab7d03b89-model-cache\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.898985 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:58.898983    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwwm\" (UniqueName: \"kubernetes.io/projected/db08db2e-17e4-426c-89d4-ae8ab7d03b89-kube-api-access-jtwwm\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:45:59.533991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:59.533922    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jgrrc"
Sep 17 17:45:59.533991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:59.533939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jgrrc" event={"ID":"db08db2e-17e4-426c-89d4-ae8ab7d03b89","Type":"ContainerDied","Data":"aef10bb614523922aea9ef2ba181793a99fa3003af0adbec31ef6bd320750248"}
Sep 17 17:45:59.533991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:45:59.533975    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aef10bb614523922aea9ef2ba181793a99fa3003af0adbec31ef6bd320750248"
Sep 17 17:46:08.499900 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.499863    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-22ff4"]
Sep 17 17:46:08.500273 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.500192    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerName="download-hf-model"
Sep 17 17:46:08.500273 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.500209    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerName="download-hf-model"
Sep 17 17:46:08.500273 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.500225    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerName="s3-init"
Sep 17 17:46:08.500273 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.500231    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerName="s3-init"
Sep 17 17:46:08.500442 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.500313    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="db08db2e-17e4-426c-89d4-ae8ab7d03b89" containerName="s3-init"
Sep 17 17:46:08.503203 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.503188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.505802 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.505771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 17:46:08.505908 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.505843    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 17:46:08.512612 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.512585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-22ff4"]
Sep 17 17:46:08.668599 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.668565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/22478116-1143-4463-90d3-2baf7abe1142-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.668599 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.668600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22478116-1143-4463-90d3-2baf7abe1142-data\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.668820 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.668631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vp9\" (UniqueName: \"kubernetes.io/projected/22478116-1143-4463-90d3-2baf7abe1142-kube-api-access-k7vp9\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.769514 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.769421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/22478116-1143-4463-90d3-2baf7abe1142-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.769514 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.769461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22478116-1143-4463-90d3-2baf7abe1142-data\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.769514 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.769501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vp9\" (UniqueName: \"kubernetes.io/projected/22478116-1143-4463-90d3-2baf7abe1142-kube-api-access-k7vp9\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.769963 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.769943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22478116-1143-4463-90d3-2baf7abe1142-data\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.771949 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.771925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/22478116-1143-4463-90d3-2baf7abe1142-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.779348 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.779314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vp9\" (UniqueName: \"kubernetes.io/projected/22478116-1143-4463-90d3-2baf7abe1142-kube-api-access-k7vp9\") pod \"seaweedfs-tls-custom-8b44d767d-22ff4\" (UID: \"22478116-1143-4463-90d3-2baf7abe1142\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:08.814161 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:08.814139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4"
Sep 17 17:46:09.083910 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.083886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-22ff4"]
Sep 17 17:46:09.086551 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:46:09.086526    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22478116_1143_4463_90d3_2baf7abe1142.slice/crio-27ae6ba5e0751da1f8c1804d1e51b54bbfa1d4f139d5d31a21fd5884742d1cf0 WatchSource:0}: Error finding container 27ae6ba5e0751da1f8c1804d1e51b54bbfa1d4f139d5d31a21fd5884742d1cf0: Status 404 returned error can't find the container with id 27ae6ba5e0751da1f8c1804d1e51b54bbfa1d4f139d5d31a21fd5884742d1cf0
Sep 17 17:46:09.577075 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.577047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4" event={"ID":"22478116-1143-4463-90d3-2baf7abe1142","Type":"ContainerStarted","Data":"0668c12dd5070e8c8ff4d95c6106d5e6964a5631eda2e494de6005566e8fa47a"}
Sep 17 17:46:09.577445 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.577090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4" event={"ID":"22478116-1143-4463-90d3-2baf7abe1142","Type":"ContainerStarted","Data":"27ae6ba5e0751da1f8c1804d1e51b54bbfa1d4f139d5d31a21fd5884742d1cf0"}
Sep 17 17:46:09.602466 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.602395    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-22ff4" podStartSLOduration=1.29674187 podStartE2EDuration="1.602377473s" totalImagesPullingTime="305.635603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:09.600558345 +0000 UTC m=+714.048388797" watchObservedRunningTime="2026-09-17 17:46:09.602377473 +0000 UTC m=+714.050207929"
Sep 17 17:46:09.922143 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.922109    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-929h4"]
Sep 17 17:46:09.925583 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.925562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:09.929159 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.929139    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:46:09.941442 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:09.941420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-929h4"]
Sep 17 17:46:10.081194 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.081162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-model-cache\") pod \"s3-tls-init-custom-929h4\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") " pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.081322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.081251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnrq\" (UniqueName: \"kubernetes.io/projected/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-kube-api-access-cdnrq\") pod \"s3-tls-init-custom-929h4\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") " pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.182254 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.182187    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnrq\" (UniqueName: \"kubernetes.io/projected/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-kube-api-access-cdnrq\") pod \"s3-tls-init-custom-929h4\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") " pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.182254 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.182237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-model-cache\") pod \"s3-tls-init-custom-929h4\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") " pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.182564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.182545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-model-cache\") pod \"s3-tls-init-custom-929h4\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") " pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.192144 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.192120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnrq\" (UniqueName: \"kubernetes.io/projected/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-kube-api-access-cdnrq\") pod \"s3-tls-init-custom-929h4\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") " pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.245046 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.245028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:10.517187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.517122    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-929h4"]
Sep 17 17:46:10.520312 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:46:10.520286    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd47ab0bf_69e5_4414_aad5_9f69615c1ebc.slice/crio-27fbcb3be8850b08fc924866169324667e03381f510a77d07092b8d3d9b6be8a WatchSource:0}: Error finding container 27fbcb3be8850b08fc924866169324667e03381f510a77d07092b8d3d9b6be8a: Status 404 returned error can't find the container with id 27fbcb3be8850b08fc924866169324667e03381f510a77d07092b8d3d9b6be8a
Sep 17 17:46:10.582708 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:10.582684    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-929h4" event={"ID":"d47ab0bf-69e5-4414-aad5-9f69615c1ebc","Type":"ContainerStarted","Data":"27fbcb3be8850b08fc924866169324667e03381f510a77d07092b8d3d9b6be8a"}
Sep 17 17:46:11.589549 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:11.589510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-929h4" event={"ID":"d47ab0bf-69e5-4414-aad5-9f69615c1ebc","Type":"ContainerStarted","Data":"7ea5d5f3cc893f3547f3c4153a55a0a0f6b7bd227be56385018e32a04ff9d3d6"}
Sep 17 17:46:12.595451 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:12.595387    2562 generic.go:356] "Generic (PLEG): container finished" podID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerID="7ea5d5f3cc893f3547f3c4153a55a0a0f6b7bd227be56385018e32a04ff9d3d6" exitCode=0
Sep 17 17:46:12.595744 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:12.595450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-929h4" event={"ID":"d47ab0bf-69e5-4414-aad5-9f69615c1ebc","Type":"ContainerDied","Data":"7ea5d5f3cc893f3547f3c4153a55a0a0f6b7bd227be56385018e32a04ff9d3d6"}
Sep 17 17:46:13.601593 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:13.601565    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-929h4" event={"ID":"d47ab0bf-69e5-4414-aad5-9f69615c1ebc","Type":"ContainerStarted","Data":"fd3d74c8fda11c5f7cc561add777605ef2c560302c6b9ce790297f760d1c7eaa"}
Sep 17 17:46:13.621856 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:13.621816    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-929h4" podStartSLOduration=2.887634548 podStartE2EDuration="4.621799108s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.73416456s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:13.621248872 +0000 UTC m=+718.069079326" watchObservedRunningTime="2026-09-17 17:46:13.621799108 +0000 UTC m=+718.069629566"
Sep 17 17:46:18.624305 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:18.624269    2562 generic.go:356] "Generic (PLEG): container finished" podID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerID="fd3d74c8fda11c5f7cc561add777605ef2c560302c6b9ce790297f760d1c7eaa" exitCode=0
Sep 17 17:46:18.624698 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:18.624323    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-929h4" event={"ID":"d47ab0bf-69e5-4414-aad5-9f69615c1ebc","Type":"ContainerDied","Data":"fd3d74c8fda11c5f7cc561add777605ef2c560302c6b9ce790297f760d1c7eaa"}
Sep 17 17:46:19.759741 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.759719    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:19.861455 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.861431    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-model-cache\") pod \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") "
Sep 17 17:46:19.861568 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.861462    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdnrq\" (UniqueName: \"kubernetes.io/projected/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-kube-api-access-cdnrq\") pod \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\" (UID: \"d47ab0bf-69e5-4414-aad5-9f69615c1ebc\") "
Sep 17 17:46:19.863464 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.863433    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-kube-api-access-cdnrq" (OuterVolumeSpecName: "kube-api-access-cdnrq") pod "d47ab0bf-69e5-4414-aad5-9f69615c1ebc" (UID: "d47ab0bf-69e5-4414-aad5-9f69615c1ebc"). InnerVolumeSpecName "kube-api-access-cdnrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:19.883023 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.882967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d47ab0bf-69e5-4414-aad5-9f69615c1ebc" (UID: "d47ab0bf-69e5-4414-aad5-9f69615c1ebc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:19.962616 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.962594    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-model-cache\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:46:19.962616 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:19.962613    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdnrq\" (UniqueName: \"kubernetes.io/projected/d47ab0bf-69e5-4414-aad5-9f69615c1ebc-kube-api-access-cdnrq\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:46:20.633452 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:20.633425    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-929h4"
Sep 17 17:46:20.633452 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:20.633437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-929h4" event={"ID":"d47ab0bf-69e5-4414-aad5-9f69615c1ebc","Type":"ContainerDied","Data":"27fbcb3be8850b08fc924866169324667e03381f510a77d07092b8d3d9b6be8a"}
Sep 17 17:46:20.633711 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:20.633468    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27fbcb3be8850b08fc924866169324667e03381f510a77d07092b8d3d9b6be8a"
Sep 17 17:46:25.330386 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.330351    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-pn8gk"]
Sep 17 17:46:25.330845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.330801    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerName="download-hf-model"
Sep 17 17:46:25.330845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.330819    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerName="download-hf-model"
Sep 17 17:46:25.330845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.330844    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerName="s3-tls-init-custom"
Sep 17 17:46:25.331014 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.330853    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerName="s3-tls-init-custom"
Sep 17 17:46:25.331014 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.330948    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d47ab0bf-69e5-4414-aad5-9f69615c1ebc" containerName="s3-tls-init-custom"
Sep 17 17:46:25.334762 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.334743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.337461 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.337444    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 17:46:25.337985 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.337971    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:46:25.348290 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.348268    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-pn8gk"]
Sep 17 17:46:25.401224 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.401200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb4956-c0a8-4a11-899b-75904645726e-model-cache\") pod \"s3-tls-init-serving-pn8gk\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") " pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.401363 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.401307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jkd\" (UniqueName: \"kubernetes.io/projected/8ecb4956-c0a8-4a11-899b-75904645726e-kube-api-access-t7jkd\") pod \"s3-tls-init-serving-pn8gk\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") " pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.501880 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.501856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jkd\" (UniqueName: \"kubernetes.io/projected/8ecb4956-c0a8-4a11-899b-75904645726e-kube-api-access-t7jkd\") pod \"s3-tls-init-serving-pn8gk\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") " pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.501984 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.501889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb4956-c0a8-4a11-899b-75904645726e-model-cache\") pod \"s3-tls-init-serving-pn8gk\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") " pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.502194 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.502178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb4956-c0a8-4a11-899b-75904645726e-model-cache\") pod \"s3-tls-init-serving-pn8gk\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") " pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.520982 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.520955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jkd\" (UniqueName: \"kubernetes.io/projected/8ecb4956-c0a8-4a11-899b-75904645726e-kube-api-access-t7jkd\") pod \"s3-tls-init-serving-pn8gk\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") " pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.659299 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.659245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:25.933707 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:25.933628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-pn8gk"]
Sep 17 17:46:25.936508 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:46:25.936482    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ecb4956_c0a8_4a11_899b_75904645726e.slice/crio-b0634b127c75ab8ec8adc603d9adc10580090bf455e2d6366bab7608d265442d WatchSource:0}: Error finding container b0634b127c75ab8ec8adc603d9adc10580090bf455e2d6366bab7608d265442d: Status 404 returned error can't find the container with id b0634b127c75ab8ec8adc603d9adc10580090bf455e2d6366bab7608d265442d
Sep 17 17:46:26.658917 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:26.658882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pn8gk" event={"ID":"8ecb4956-c0a8-4a11-899b-75904645726e","Type":"ContainerStarted","Data":"c1f979569b1bcd28e3bd2aca242a664888028576a6076a09daf34a1f9f92d3dc"}
Sep 17 17:46:26.659259 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:26.658928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pn8gk" event={"ID":"8ecb4956-c0a8-4a11-899b-75904645726e","Type":"ContainerStarted","Data":"b0634b127c75ab8ec8adc603d9adc10580090bf455e2d6366bab7608d265442d"}
Sep 17 17:46:27.667659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:27.667630    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ecb4956-c0a8-4a11-899b-75904645726e" containerID="c1f979569b1bcd28e3bd2aca242a664888028576a6076a09daf34a1f9f92d3dc" exitCode=0
Sep 17 17:46:27.667974 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:27.667710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pn8gk" event={"ID":"8ecb4956-c0a8-4a11-899b-75904645726e","Type":"ContainerDied","Data":"c1f979569b1bcd28e3bd2aca242a664888028576a6076a09daf34a1f9f92d3dc"}
Sep 17 17:46:28.673702 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:28.673664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pn8gk" event={"ID":"8ecb4956-c0a8-4a11-899b-75904645726e","Type":"ContainerStarted","Data":"e219c78926a12f7980e01de77bfe6e98ff38a9574ce4dcfe25f91ff5d88998d5"}
Sep 17 17:46:28.707630 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:28.707584    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-pn8gk" podStartSLOduration=2.079654328 podStartE2EDuration="3.707570929s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.627916601s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:28.705106104 +0000 UTC m=+733.152936557" watchObservedRunningTime="2026-09-17 17:46:28.707570929 +0000 UTC m=+733.155401406"
Sep 17 17:46:34.705324 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:34.705289    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ecb4956-c0a8-4a11-899b-75904645726e" containerID="e219c78926a12f7980e01de77bfe6e98ff38a9574ce4dcfe25f91ff5d88998d5" exitCode=0
Sep 17 17:46:34.705714 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:34.705356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pn8gk" event={"ID":"8ecb4956-c0a8-4a11-899b-75904645726e","Type":"ContainerDied","Data":"e219c78926a12f7980e01de77bfe6e98ff38a9574ce4dcfe25f91ff5d88998d5"}
Sep 17 17:46:35.845138 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.845115    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:35.887410 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.887388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb4956-c0a8-4a11-899b-75904645726e-model-cache\") pod \"8ecb4956-c0a8-4a11-899b-75904645726e\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") "
Sep 17 17:46:35.887539 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.887424    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7jkd\" (UniqueName: \"kubernetes.io/projected/8ecb4956-c0a8-4a11-899b-75904645726e-kube-api-access-t7jkd\") pod \"8ecb4956-c0a8-4a11-899b-75904645726e\" (UID: \"8ecb4956-c0a8-4a11-899b-75904645726e\") "
Sep 17 17:46:35.889344 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.889309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecb4956-c0a8-4a11-899b-75904645726e-kube-api-access-t7jkd" (OuterVolumeSpecName: "kube-api-access-t7jkd") pod "8ecb4956-c0a8-4a11-899b-75904645726e" (UID: "8ecb4956-c0a8-4a11-899b-75904645726e"). InnerVolumeSpecName "kube-api-access-t7jkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:35.909847 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.909821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb4956-c0a8-4a11-899b-75904645726e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ecb4956-c0a8-4a11-899b-75904645726e" (UID: "8ecb4956-c0a8-4a11-899b-75904645726e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:35.988863 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.988810    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb4956-c0a8-4a11-899b-75904645726e-model-cache\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:46:35.988863 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:35.988833    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7jkd\" (UniqueName: \"kubernetes.io/projected/8ecb4956-c0a8-4a11-899b-75904645726e-kube-api-access-t7jkd\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:46:36.715668 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:36.715645    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pn8gk"
Sep 17 17:46:36.715858 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:36.715646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pn8gk" event={"ID":"8ecb4956-c0a8-4a11-899b-75904645726e","Type":"ContainerDied","Data":"b0634b127c75ab8ec8adc603d9adc10580090bf455e2d6366bab7608d265442d"}
Sep 17 17:46:36.715858 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:36.715750    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0634b127c75ab8ec8adc603d9adc10580090bf455e2d6366bab7608d265442d"
Sep 17 17:46:48.995087 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:48.995051    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"]
Sep 17 17:46:48.995805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:48.995550    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb4956-c0a8-4a11-899b-75904645726e" containerName="s3-tls-init-serving"
Sep 17 17:46:48.995805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:48.995569    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb4956-c0a8-4a11-899b-75904645726e" containerName="s3-tls-init-serving"
Sep 17 17:46:48.995805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:48.995582    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb4956-c0a8-4a11-899b-75904645726e" containerName="download-hf-model"
Sep 17 17:46:48.995805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:48.995590    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb4956-c0a8-4a11-899b-75904645726e" containerName="download-hf-model"
Sep 17 17:46:48.995805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:48.995679    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecb4956-c0a8-4a11-899b-75904645726e" containerName="s3-tls-init-serving"
Sep 17 17:46:49.000308 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.000287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.003968 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.003949    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 17:46:49.004173 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.004151    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:46:49.004307 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.004287    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:46:49.004563 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.004313    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:46:49.004563 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.004414    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:46:49.005059 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.005044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 17:46:49.005179 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.005161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:46:49.010290 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.010270    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"]
Sep 17 17:46:49.092939 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.092912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9vn8\" (UniqueName: \"kubernetes.io/projected/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-api-access-c9vn8\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.092939 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.092945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.093132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.092991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.093132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.093062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.093132 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.093109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.193635 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.193606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9vn8\" (UniqueName: \"kubernetes.io/projected/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-api-access-c9vn8\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.193797 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.193649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.193797 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.193677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.193797 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.193739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.193797 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.193776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.194147 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.194124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.194454 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.194437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.194519 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.194468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.195960 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.195928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.209144 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.209118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9vn8\" (UniqueName: \"kubernetes.io/projected/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-api-access-c9vn8\") pod \"isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.295939 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.295888    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"]
Sep 17 17:46:49.299639 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.299626    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.302544 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.302523    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-8009d5-kube-rbac-proxy-sar-config\""
Sep 17 17:46:49.302642 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.302581    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-8009d5-predictor-serving-cert\""
Sep 17 17:46:49.302701 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.302649    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-8009d5-dockercfg-p9dfq\""
Sep 17 17:46:49.311703 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.311684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:46:49.317589 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.317572    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-8009d5\""
Sep 17 17:46:49.324993 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.324971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"]
Sep 17 17:46:49.394829 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.394797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894hg\" (UniqueName: \"kubernetes.io/projected/04593d34-4770-4513-bee6-fe6f5482d611-kube-api-access-894hg\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.394946 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.394852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04593d34-4770-4513-bee6-fe6f5482d611-kserve-provision-location\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.394946 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.394915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-cabundle-cert\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.395021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.394952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04593d34-4770-4513-bee6-fe6f5482d611-proxy-tls\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.395021 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.394984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.495609 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.495586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04593d34-4770-4513-bee6-fe6f5482d611-kserve-provision-location\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.495725 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.495619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-cabundle-cert\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.495725 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.495646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04593d34-4770-4513-bee6-fe6f5482d611-proxy-tls\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.495725 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.495674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.495725 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.495703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894hg\" (UniqueName: \"kubernetes.io/projected/04593d34-4770-4513-bee6-fe6f5482d611-kube-api-access-894hg\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.496028 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.496007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04593d34-4770-4513-bee6-fe6f5482d611-kserve-provision-location\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.496287 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.496266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-cabundle-cert\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.496393 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.496355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.497943 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.497920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04593d34-4770-4513-bee6-fe6f5482d611-proxy-tls\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.506457 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.506437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894hg\" (UniqueName: \"kubernetes.io/projected/04593d34-4770-4513-bee6-fe6f5482d611-kube-api-access-894hg\") pod \"isvc-init-fail-8009d5-predictor-549687fbc6-9758s\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") " pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.595583 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.595561    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"]
Sep 17 17:46:49.597275 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:46:49.597252    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2f33b8_5c1f_425b_a8d0_8fc3c74a05c9.slice/crio-9b2d61c6f41a12621040b9c4b4d1cfa2a254de3effeac873f79710f6c9f0b178 WatchSource:0}: Error finding container 9b2d61c6f41a12621040b9c4b4d1cfa2a254de3effeac873f79710f6c9f0b178: Status 404 returned error can't find the container with id 9b2d61c6f41a12621040b9c4b4d1cfa2a254de3effeac873f79710f6c9f0b178
Sep 17 17:46:49.609522 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.609505    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:46:49.768791 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.768760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerStarted","Data":"789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11"}
Sep 17 17:46:49.768943 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.768817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerStarted","Data":"9b2d61c6f41a12621040b9c4b4d1cfa2a254de3effeac873f79710f6c9f0b178"}
Sep 17 17:46:49.883583 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:49.883561    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"]
Sep 17 17:46:49.886297 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:46:49.886265    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04593d34_4770_4513_bee6_fe6f5482d611.slice/crio-5992770682cbbd229f970339200ddaf726891f5b3eceaadc5f8b502355ea1a3b WatchSource:0}: Error finding container 5992770682cbbd229f970339200ddaf726891f5b3eceaadc5f8b502355ea1a3b: Status 404 returned error can't find the container with id 5992770682cbbd229f970339200ddaf726891f5b3eceaadc5f8b502355ea1a3b
Sep 17 17:46:50.776324 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:50.776289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" event={"ID":"04593d34-4770-4513-bee6-fe6f5482d611","Type":"ContainerStarted","Data":"87e2b2588a4e6c7f6e8ae6ffb27d15e8becad0d97e91b31cc31297aa939a5ba7"}
Sep 17 17:46:50.776759 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:50.776364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" event={"ID":"04593d34-4770-4513-bee6-fe6f5482d611","Type":"ContainerStarted","Data":"5992770682cbbd229f970339200ddaf726891f5b3eceaadc5f8b502355ea1a3b"}
Sep 17 17:46:53.791091 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:53.791052    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerID="789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11" exitCode=0
Sep 17 17:46:53.791494 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:53.791124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerDied","Data":"789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11"}
Sep 17 17:46:55.810125 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:55.810097    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8009d5-predictor-549687fbc6-9758s_04593d34-4770-4513-bee6-fe6f5482d611/storage-initializer/0.log"
Sep 17 17:46:55.810646 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:55.810143    2562 generic.go:356] "Generic (PLEG): container finished" podID="04593d34-4770-4513-bee6-fe6f5482d611" containerID="87e2b2588a4e6c7f6e8ae6ffb27d15e8becad0d97e91b31cc31297aa939a5ba7" exitCode=1
Sep 17 17:46:55.810646 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:55.810252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" event={"ID":"04593d34-4770-4513-bee6-fe6f5482d611","Type":"ContainerDied","Data":"87e2b2588a4e6c7f6e8ae6ffb27d15e8becad0d97e91b31cc31297aa939a5ba7"}
Sep 17 17:46:56.821618 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:56.821455    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8009d5-predictor-549687fbc6-9758s_04593d34-4770-4513-bee6-fe6f5482d611/storage-initializer/0.log"
Sep 17 17:46:56.821618 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:56.821563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" event={"ID":"04593d34-4770-4513-bee6-fe6f5482d611","Type":"ContainerStarted","Data":"7e0b9404f5c53b89ae814e420daa8247b2f1ef82be5a77ac99b3a54da41c8862"}
Sep 17 17:46:58.799870 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:58.799753    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"]
Sep 17 17:46:58.800386 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:46:58.799990    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer" containerID="cri-o://7e0b9404f5c53b89ae814e420daa8247b2f1ef82be5a77ac99b3a54da41c8862" gracePeriod=30
Sep 17 17:47:02.859731 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:02.859696    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8009d5-predictor-549687fbc6-9758s_04593d34-4770-4513-bee6-fe6f5482d611/storage-initializer/1.log"
Sep 17 17:47:02.860272 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:02.860112    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8009d5-predictor-549687fbc6-9758s_04593d34-4770-4513-bee6-fe6f5482d611/storage-initializer/0.log"
Sep 17 17:47:02.860272 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:02.860143    2562 generic.go:356] "Generic (PLEG): container finished" podID="04593d34-4770-4513-bee6-fe6f5482d611" containerID="7e0b9404f5c53b89ae814e420daa8247b2f1ef82be5a77ac99b3a54da41c8862" exitCode=1
Sep 17 17:47:02.860272 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:02.860205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" event={"ID":"04593d34-4770-4513-bee6-fe6f5482d611","Type":"ContainerDied","Data":"7e0b9404f5c53b89ae814e420daa8247b2f1ef82be5a77ac99b3a54da41c8862"}
Sep 17 17:47:02.860272 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:02.860264    2562 scope.go:117] "RemoveContainer" containerID="87e2b2588a4e6c7f6e8ae6ffb27d15e8becad0d97e91b31cc31297aa939a5ba7"
Sep 17 17:47:06.319117 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.319096    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8009d5-predictor-549687fbc6-9758s_04593d34-4770-4513-bee6-fe6f5482d611/storage-initializer/1.log"
Sep 17 17:47:06.321254 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.321238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:47:06.449549 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.449524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-894hg\" (UniqueName: \"kubernetes.io/projected/04593d34-4770-4513-bee6-fe6f5482d611-kube-api-access-894hg\") pod \"04593d34-4770-4513-bee6-fe6f5482d611\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") "
Sep 17 17:47:06.449696 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.449579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-cabundle-cert\") pod \"04593d34-4770-4513-bee6-fe6f5482d611\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") "
Sep 17 17:47:06.449696 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.449627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04593d34-4770-4513-bee6-fe6f5482d611-kserve-provision-location\") pod \"04593d34-4770-4513-bee6-fe6f5482d611\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") "
Sep 17 17:47:06.449696 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.449662    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-kube-rbac-proxy-sar-config\") pod \"04593d34-4770-4513-bee6-fe6f5482d611\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") "
Sep 17 17:47:06.449883 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.449719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04593d34-4770-4513-bee6-fe6f5482d611-proxy-tls\") pod \"04593d34-4770-4513-bee6-fe6f5482d611\" (UID: \"04593d34-4770-4513-bee6-fe6f5482d611\") "
Sep 17 17:47:06.449963 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.449934    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04593d34-4770-4513-bee6-fe6f5482d611-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04593d34-4770-4513-bee6-fe6f5482d611" (UID: "04593d34-4770-4513-bee6-fe6f5482d611"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:06.450029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.450013    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04593d34-4770-4513-bee6-fe6f5482d611" (UID: "04593d34-4770-4513-bee6-fe6f5482d611"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:06.450079 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.450027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04593d34-4770-4513-bee6-fe6f5482d611" (UID: "04593d34-4770-4513-bee6-fe6f5482d611"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:06.450079 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.450053    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04593d34-4770-4513-bee6-fe6f5482d611-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:47:06.451681 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.451650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04593d34-4770-4513-bee6-fe6f5482d611-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04593d34-4770-4513-bee6-fe6f5482d611" (UID: "04593d34-4770-4513-bee6-fe6f5482d611"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:06.451802 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.451784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04593d34-4770-4513-bee6-fe6f5482d611-kube-api-access-894hg" (OuterVolumeSpecName: "kube-api-access-894hg") pod "04593d34-4770-4513-bee6-fe6f5482d611" (UID: "04593d34-4770-4513-bee6-fe6f5482d611"). InnerVolumeSpecName "kube-api-access-894hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:06.550901 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.550870    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:47:06.550901 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.550896    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04593d34-4770-4513-bee6-fe6f5482d611-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:47:06.550901 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.550907    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894hg\" (UniqueName: \"kubernetes.io/projected/04593d34-4770-4513-bee6-fe6f5482d611-kube-api-access-894hg\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:47:06.551139 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.550918    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04593d34-4770-4513-bee6-fe6f5482d611-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:47:06.885163 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.885122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerStarted","Data":"964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe"}
Sep 17 17:47:06.886763 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.886720    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8009d5-predictor-549687fbc6-9758s_04593d34-4770-4513-bee6-fe6f5482d611/storage-initializer/1.log"
Sep 17 17:47:06.886892 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.886804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s" event={"ID":"04593d34-4770-4513-bee6-fe6f5482d611","Type":"ContainerDied","Data":"5992770682cbbd229f970339200ddaf726891f5b3eceaadc5f8b502355ea1a3b"}
Sep 17 17:47:06.886892 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.886814    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"
Sep 17 17:47:06.886892 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.886855    2562 scope.go:117] "RemoveContainer" containerID="7e0b9404f5c53b89ae814e420daa8247b2f1ef82be5a77ac99b3a54da41c8862"
Sep 17 17:47:06.931719 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.931664    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"]
Sep 17 17:47:06.938179 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:06.938120    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8009d5-predictor-549687fbc6-9758s"]
Sep 17 17:47:07.728369 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:07.728325    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04593d34-4770-4513-bee6-fe6f5482d611" path="/var/lib/kubelet/pods/04593d34-4770-4513-bee6-fe6f5482d611/volumes"
Sep 17 17:47:08.899596 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:08.899518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerStarted","Data":"9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c"}
Sep 17 17:47:11.918925 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:11.918880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerStarted","Data":"8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73"}
Sep 17 17:47:11.919393 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:11.919119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:47:11.919393 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:11.919293    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:47:11.920691 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:11.920662    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:11.948212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:11.948166    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podStartSLOduration=2.65470394 podStartE2EDuration="23.948153138s" totalImagesPullingTime="17.309780718s" totalInitContainerRuntime="3.98366848s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:11.946601842 +0000 UTC m=+776.394432335" watchObservedRunningTime="2026-09-17 17:47:11.948153138 +0000 UTC m=+776.395983593"
Sep 17 17:47:12.923548 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:12.923511    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:47:12.924012 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:12.923674    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:12.924888 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:12.924857    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:47:13.928362 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:13.928304    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:13.928806 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:13.928675    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:47:13.932142 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:13.932121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:47:14.932525 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:14.932487    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:14.932956 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:14.932912    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:47:24.933026 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:24.932977    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:24.933545 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:24.933461    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:47:34.932715 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:34.932628    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:34.933180 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:34.933062    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:47:44.933430 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:44.933379    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:44.933968 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:44.933882    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:47:54.933143 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:54.933039    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:47:54.933614 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:47:54.933569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:04.932937 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:04.932885    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:48:04.933366 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:04.933294    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:14.933446 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:14.933398    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:48:14.933927 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:14.933786    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:24.933515 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:24.933482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:48:24.933882 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:24.933577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:48:33.926041 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:33.926008    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"]
Sep 17 17:48:33.926459 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:33.926275    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" containerID="cri-o://964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe" gracePeriod=30
Sep 17 17:48:33.926459 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:33.926314    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" containerID="cri-o://9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c" gracePeriod=30
Sep 17 17:48:33.926459 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:33.926307    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" containerID="cri-o://8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73" gracePeriod=30
Sep 17 17:48:33.931471 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:33.931440    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": read tcp 10.132.0.2:49288->10.132.0.28:8643: read: connection reset by peer"
Sep 17 17:48:34.083487 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.083454    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"]
Sep 17 17:48:34.083840 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.083825    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer"
Sep 17 17:48:34.083887 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.083842    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer"
Sep 17 17:48:34.083887 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.083853    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer"
Sep 17 17:48:34.083887 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.083858    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer"
Sep 17 17:48:34.083994 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.083913    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer"
Sep 17 17:48:34.084029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.084023    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04593d34-4770-4513-bee6-fe6f5482d611" containerName="storage-initializer"
Sep 17 17:48:34.087310 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.087293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.091382 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.091361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 17:48:34.091565 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.091371    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 17:48:34.107182 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.107156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"]
Sep 17 17:48:34.194275 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.194185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7hn\" (UniqueName: \"kubernetes.io/projected/8eb4058a-4006-48a4-9322-c5e553f94532-kube-api-access-wb7hn\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.194275 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.194235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.194486 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.194313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.194486 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.194407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eb4058a-4006-48a4-9322-c5e553f94532-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.194486 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.194480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eb4058a-4006-48a4-9322-c5e553f94532-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.295450 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.295416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eb4058a-4006-48a4-9322-c5e553f94532-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.295615 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.295463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7hn\" (UniqueName: \"kubernetes.io/projected/8eb4058a-4006-48a4-9322-c5e553f94532-kube-api-access-wb7hn\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.295615 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.295482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.295615 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.295513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.295615 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.295549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eb4058a-4006-48a4-9322-c5e553f94532-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.296094 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.296071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eb4058a-4006-48a4-9322-c5e553f94532-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.296320 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.296298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.296404 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.296319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.297855 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.297835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eb4058a-4006-48a4-9322-c5e553f94532-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.305952 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.305923    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerID="9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c" exitCode=2
Sep 17 17:48:34.306049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.305988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerDied","Data":"9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c"}
Sep 17 17:48:34.307278 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.307258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7hn\" (UniqueName: \"kubernetes.io/projected/8eb4058a-4006-48a4-9322-c5e553f94532-kube-api-access-wb7hn\") pod \"isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.397483 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.397456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:34.688936 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.688911    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"]
Sep 17 17:48:34.932765 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.932711    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:48:34.933170 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:34.933032    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:35.313627 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:35.313596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerStarted","Data":"eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a"}
Sep 17 17:48:35.313866 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:35.313678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerStarted","Data":"92d0e15b0f8612dfc5f3247f5dbfdaf4389d2b0e55461a57d40a497444abb503"}
Sep 17 17:48:38.929033 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:38.928991    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:48:39.333973 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:39.333941    2562 generic.go:356] "Generic (PLEG): container finished" podID="8eb4058a-4006-48a4-9322-c5e553f94532" containerID="eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a" exitCode=0
Sep 17 17:48:39.334116 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:39.334012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerDied","Data":"eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a"}
Sep 17 17:48:39.336861 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:39.336840    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerID="964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe" exitCode=0
Sep 17 17:48:39.336949 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:39.336889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerDied","Data":"964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe"}
Sep 17 17:48:40.343645 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:40.343611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerStarted","Data":"34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a"}
Sep 17 17:48:40.344041 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:40.343700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerStarted","Data":"081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b"}
Sep 17 17:48:40.344041 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:40.343727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerStarted","Data":"589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219"}
Sep 17 17:48:40.344041 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:40.343936    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:40.377571 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:40.377527    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podStartSLOduration=2.6352754689999998 podStartE2EDuration="6.377511148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742235679s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:40.374033832 +0000 UTC m=+864.821864320" watchObservedRunningTime="2026-09-17 17:48:40.377511148 +0000 UTC m=+864.825341603"
Sep 17 17:48:41.347964 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:41.347925    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:41.348406 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:41.347989    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:41.349419 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:41.349387    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:48:41.350039 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:41.350010    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:42.352196 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:42.352157    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:48:42.352650 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:42.352627    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:43.929178 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:43.929138    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:48:43.929575 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:43.929279    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:48:44.932912 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:44.932871    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:48:44.933395 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:44.933193    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:47.357513 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:47.357486    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:48:47.358086 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:47.358001    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:48:47.358381 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:47.358342    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:48.929217 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:48.929169    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:48:53.929293 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:53.929251    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:48:54.932597 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:54.932502    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 17:48:54.932977 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:54.932643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:48:54.932977 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:54.932812    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:54.932977 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:54.932911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:48:57.358459 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:57.358423    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:48:57.358867 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:57.358843    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:58.929231 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:48:58.929185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:49:03.929386 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:03.929315    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 17:49:04.083230 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.083203    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:49:04.144738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.144700    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-proxy-tls\") pod \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") "
Sep 17 17:49:04.144738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.144746    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9vn8\" (UniqueName: \"kubernetes.io/projected/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-api-access-c9vn8\") pod \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") "
Sep 17 17:49:04.144986 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.144833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kserve-provision-location\") pod \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") "
Sep 17 17:49:04.144986 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.144862    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-rbac-proxy-sar-config\") pod \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") "
Sep 17 17:49:04.144986 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.144882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-cabundle-cert\") pod \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\" (UID: \"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9\") "
Sep 17 17:49:04.145245 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.145216    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" (UID: "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:04.145245 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.145235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" (UID: "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:04.145464 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.145269    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" (UID: "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:04.146977 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.146949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" (UID: "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:04.147062 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.146977    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-api-access-c9vn8" (OuterVolumeSpecName: "kube-api-access-c9vn8") pod "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" (UID: "1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9"). InnerVolumeSpecName "kube-api-access-c9vn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:04.245619 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.245528    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:49:04.245619 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.245558    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:49:04.245619 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.245569    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:49:04.245619 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.245581    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:49:04.245619 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.245589    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9vn8\" (UniqueName: \"kubernetes.io/projected/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9-kube-api-access-c9vn8\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:49:04.462067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.462036    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerID="8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73" exitCode=0
Sep 17 17:49:04.462257 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.462104    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"
Sep 17 17:49:04.462257 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.462116    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerDied","Data":"8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73"}
Sep 17 17:49:04.462257 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.462180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh" event={"ID":"1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9","Type":"ContainerDied","Data":"9b2d61c6f41a12621040b9c4b4d1cfa2a254de3effeac873f79710f6c9f0b178"}
Sep 17 17:49:04.462257 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.462213    2562 scope.go:117] "RemoveContainer" containerID="8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73"
Sep 17 17:49:04.471149 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.471134    2562 scope.go:117] "RemoveContainer" containerID="9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c"
Sep 17 17:49:04.478726 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.478709    2562 scope.go:117] "RemoveContainer" containerID="964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe"
Sep 17 17:49:04.486638 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.486614    2562 scope.go:117] "RemoveContainer" containerID="789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11"
Sep 17 17:49:04.487715 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.487695    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"]
Sep 17 17:49:04.489250 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.489226    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7485d8d7d4-8gmjh"]
Sep 17 17:49:04.494588 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.494574    2562 scope.go:117] "RemoveContainer" containerID="8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73"
Sep 17 17:49:04.494862 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:49:04.494845    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73\": container with ID starting with 8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73 not found: ID does not exist" containerID="8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73"
Sep 17 17:49:04.494914 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.494870    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73"} err="failed to get container status \"8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73\": rpc error: code = NotFound desc = could not find container \"8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73\": container with ID starting with 8ffb394a885bd49872b55e1f15746ca574a30b272b011c9583770e1a9ba80e73 not found: ID does not exist"
Sep 17 17:49:04.494914 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.494900    2562 scope.go:117] "RemoveContainer" containerID="9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c"
Sep 17 17:49:04.495145 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:49:04.495128    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c\": container with ID starting with 9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c not found: ID does not exist" containerID="9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c"
Sep 17 17:49:04.495212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.495157    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c"} err="failed to get container status \"9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c\": rpc error: code = NotFound desc = could not find container \"9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c\": container with ID starting with 9a4cc43f0f869137f6f437240e5d76b188d69332d228b73d00564a09fa3d6a8c not found: ID does not exist"
Sep 17 17:49:04.495212 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.495179    2562 scope.go:117] "RemoveContainer" containerID="964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe"
Sep 17 17:49:04.495479 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:49:04.495459    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe\": container with ID starting with 964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe not found: ID does not exist" containerID="964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe"
Sep 17 17:49:04.495524 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.495486    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe"} err="failed to get container status \"964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe\": rpc error: code = NotFound desc = could not find container \"964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe\": container with ID starting with 964d443fa7ecc473f594e64355b7120215fc4bfeaf28373ae0660900f3e389fe not found: ID does not exist"
Sep 17 17:49:04.495524 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.495502    2562 scope.go:117] "RemoveContainer" containerID="789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11"
Sep 17 17:49:04.495730 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:49:04.495692    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11\": container with ID starting with 789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11 not found: ID does not exist" containerID="789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11"
Sep 17 17:49:04.495730 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:04.495715    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11"} err="failed to get container status \"789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11\": rpc error: code = NotFound desc = could not find container \"789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11\": container with ID starting with 789c94d1cfeb287a4fb5eaf7b63f12669ee6d635d347afa75617593511ee6a11 not found: ID does not exist"
Sep 17 17:49:05.727807 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:05.727773    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" path="/var/lib/kubelet/pods/1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9/volumes"
Sep 17 17:49:07.358116 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:07.358075    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:49:07.358532 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:07.358498    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:17.358009 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:17.357962    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:49:17.358432 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:17.358392    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:17.685240 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:17.685212    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:49:17.689385 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:17.689364    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:49:27.358603 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:27.358558    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:49:27.359127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:27.358975    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:37.357972 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:37.357932    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:49:37.358464 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:37.358438    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:47.358849 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:47.358805    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:49:47.359285 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:47.359264    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:51.722481 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:51.722426    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:49:51.723040 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:49:51.723004    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:01.726568 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:01.726540    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:01.729108 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:01.726630    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:09.121495 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:09.121464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"]
Sep 17 17:50:09.121930 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:09.121738    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" containerID="cri-o://589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219" gracePeriod=30
Sep 17 17:50:09.121930 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:09.121846    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" containerID="cri-o://34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a" gracePeriod=30
Sep 17 17:50:09.121930 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:09.121842    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" containerID="cri-o://081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b" gracePeriod=30
Sep 17 17:50:09.771317 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:09.771282    2562 generic.go:356] "Generic (PLEG): container finished" podID="8eb4058a-4006-48a4-9322-c5e553f94532" containerID="081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b" exitCode=2
Sep 17 17:50:09.771538 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:09.771367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerDied","Data":"081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b"}
Sep 17 17:50:11.722965 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:11.722914    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:50:11.723405 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:11.723282    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:12.353300 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:12.353256    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:50:14.796945 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:14.796912    2562 generic.go:356] "Generic (PLEG): container finished" podID="8eb4058a-4006-48a4-9322-c5e553f94532" containerID="589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219" exitCode=0
Sep 17 17:50:14.797315 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:14.796966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerDied","Data":"589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219"}
Sep 17 17:50:17.353399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:17.353323    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:50:19.294344 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294306    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"]
Sep 17 17:50:19.294723 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294697    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent"
Sep 17 17:50:19.294723 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294712    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent"
Sep 17 17:50:19.294792 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294725    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="storage-initializer"
Sep 17 17:50:19.294792 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294731    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="storage-initializer"
Sep 17 17:50:19.294792 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294745    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container"
Sep 17 17:50:19.294792 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294750    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container"
Sep 17 17:50:19.294792 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294763    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy"
Sep 17 17:50:19.294792 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294768    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy"
Sep 17 17:50:19.294966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294818    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kserve-container"
Sep 17 17:50:19.294966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294824    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="kube-rbac-proxy"
Sep 17 17:50:19.294966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.294833    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2f33b8-5c1f-425b-a8d0-8fc3c74a05c9" containerName="agent"
Sep 17 17:50:19.298088 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.298066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.301035 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.301014    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 17:50:19.301153 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.301041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 17:50:19.313991 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.313963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"]
Sep 17 17:50:19.380627 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.380592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-cabundle-cert\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.380799 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.380648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.380799 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.380701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.380799 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.380740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjjz\" (UniqueName: \"kubernetes.io/projected/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-api-access-9tjjz\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.380799 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.380760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a04de597-9614-4eea-91db-1bfc009c0ab0-kserve-provision-location\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.481775 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.481740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-cabundle-cert\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.481971 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.481796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.481971 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.481832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.481971 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.481866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjjz\" (UniqueName: \"kubernetes.io/projected/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-api-access-9tjjz\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.481971 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.481889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a04de597-9614-4eea-91db-1bfc009c0ab0-kserve-provision-location\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.482135 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:50:19.482115    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 17 17:50:19.482211 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:50:19.482193    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls podName:a04de597-9614-4eea-91db-1bfc009c0ab0 nodeName:}" failed. No retries permitted until 2026-09-17 17:50:19.982173389 +0000 UTC m=+964.430003836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls") pod "isvc-logger-predictor-6bdfd88f5f-sqg99" (UID: "a04de597-9614-4eea-91db-1bfc009c0ab0") : secret "isvc-logger-predictor-serving-cert" not found
Sep 17 17:50:19.482543 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.482519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a04de597-9614-4eea-91db-1bfc009c0ab0-kserve-provision-location\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.482623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.482547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-cabundle-cert\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.482788 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.482765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.493282 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.493251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjjz\" (UniqueName: \"kubernetes.io/projected/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-api-access-9tjjz\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.987431 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.987394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:19.989705 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:19.989684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls\") pod \"isvc-logger-predictor-6bdfd88f5f-sqg99\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:20.208482 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:20.208446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:20.491388 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:20.491365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"]
Sep 17 17:50:20.493356 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:50:20.493301    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04de597_9614_4eea_91db_1bfc009c0ab0.slice/crio-fa0083163fbffd791fe264283a257ff58f3bc89f9b1b3470999eae9847ab5995 WatchSource:0}: Error finding container fa0083163fbffd791fe264283a257ff58f3bc89f9b1b3470999eae9847ab5995: Status 404 returned error can't find the container with id fa0083163fbffd791fe264283a257ff58f3bc89f9b1b3470999eae9847ab5995
Sep 17 17:50:20.826477 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:20.826443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerStarted","Data":"11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af"}
Sep 17 17:50:20.826653 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:20.826502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerStarted","Data":"fa0083163fbffd791fe264283a257ff58f3bc89f9b1b3470999eae9847ab5995"}
Sep 17 17:50:21.723057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:21.722998    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:50:21.723539 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:21.723396    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:22.352747 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:22.352699    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:50:22.352913 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:22.352883    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:24.845698 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:24.845667    2562 generic.go:356] "Generic (PLEG): container finished" podID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerID="11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af" exitCode=0
Sep 17 17:50:24.846083 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:24.845740    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerDied","Data":"11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af"}
Sep 17 17:50:25.852560 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.852527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerStarted","Data":"52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d"}
Sep 17 17:50:25.852928 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.852592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerStarted","Data":"e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a"}
Sep 17 17:50:25.852928 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.852617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerStarted","Data":"26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85"}
Sep 17 17:50:25.853011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.852935    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:25.853011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.852981    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:25.854415 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.854385    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:50:25.888300 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:25.888227    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podStartSLOduration=3.048634808 podStartE2EDuration="6.888209684s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.839574876s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:25.88555535 +0000 UTC m=+970.333385850" watchObservedRunningTime="2026-09-17 17:50:25.888209684 +0000 UTC m=+970.336040139"
Sep 17 17:50:26.860735 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:26.860699    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:26.861142 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:26.860830    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:50:26.861770 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:26.861746    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:27.352679 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:27.352639    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:50:27.865570 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:27.865524    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:50:27.865993 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:27.865885    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:31.723057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:31.722952    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:5000: connect: connection refused"
Sep 17 17:50:31.723506 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:31.723321    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:31.727102 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:31.727081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:31.727213 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:31.727152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:32.353081 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:32.353043    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:50:32.869759 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:32.869733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:50:32.870374 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:32.870322    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:50:32.870715 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:32.870692    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:37.352473 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:37.352432    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 17:50:39.275764 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.275741    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:39.353557 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.353525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb7hn\" (UniqueName: \"kubernetes.io/projected/8eb4058a-4006-48a4-9322-c5e553f94532-kube-api-access-wb7hn\") pod \"8eb4058a-4006-48a4-9322-c5e553f94532\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") "
Sep 17 17:50:39.353733 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.353570    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-cabundle-cert\") pod \"8eb4058a-4006-48a4-9322-c5e553f94532\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") "
Sep 17 17:50:39.353733 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.353607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eb4058a-4006-48a4-9322-c5e553f94532-proxy-tls\") pod \"8eb4058a-4006-48a4-9322-c5e553f94532\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") "
Sep 17 17:50:39.353733 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.353634    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-kube-rbac-proxy-sar-config\") pod \"8eb4058a-4006-48a4-9322-c5e553f94532\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") "
Sep 17 17:50:39.353733 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.353685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eb4058a-4006-48a4-9322-c5e553f94532-kserve-provision-location\") pod \"8eb4058a-4006-48a4-9322-c5e553f94532\" (UID: \"8eb4058a-4006-48a4-9322-c5e553f94532\") "
Sep 17 17:50:39.353940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.353913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8eb4058a-4006-48a4-9322-c5e553f94532" (UID: "8eb4058a-4006-48a4-9322-c5e553f94532"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:39.354032 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.354006    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8eb4058a-4006-48a4-9322-c5e553f94532" (UID: "8eb4058a-4006-48a4-9322-c5e553f94532"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:39.354032 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.354021    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb4058a-4006-48a4-9322-c5e553f94532-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8eb4058a-4006-48a4-9322-c5e553f94532" (UID: "8eb4058a-4006-48a4-9322-c5e553f94532"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:39.355657 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.355635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb4058a-4006-48a4-9322-c5e553f94532-kube-api-access-wb7hn" (OuterVolumeSpecName: "kube-api-access-wb7hn") pod "8eb4058a-4006-48a4-9322-c5e553f94532" (UID: "8eb4058a-4006-48a4-9322-c5e553f94532"). InnerVolumeSpecName "kube-api-access-wb7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:39.355732 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.355670    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb4058a-4006-48a4-9322-c5e553f94532-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8eb4058a-4006-48a4-9322-c5e553f94532" (UID: "8eb4058a-4006-48a4-9322-c5e553f94532"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:39.454288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.454232    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7hn\" (UniqueName: \"kubernetes.io/projected/8eb4058a-4006-48a4-9322-c5e553f94532-kube-api-access-wb7hn\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:50:39.454288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.454253    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:50:39.454288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.454263    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8eb4058a-4006-48a4-9322-c5e553f94532-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:50:39.454288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.454271    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8eb4058a-4006-48a4-9322-c5e553f94532-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:50:39.454288 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.454279    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8eb4058a-4006-48a4-9322-c5e553f94532-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:50:39.926723 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.926683    2562 generic.go:356] "Generic (PLEG): container finished" podID="8eb4058a-4006-48a4-9322-c5e553f94532" containerID="34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a" exitCode=0
Sep 17 17:50:39.926884 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.926745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"
Sep 17 17:50:39.926884 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.926773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerDied","Data":"34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a"}
Sep 17 17:50:39.926884 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.926842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m" event={"ID":"8eb4058a-4006-48a4-9322-c5e553f94532","Type":"ContainerDied","Data":"92d0e15b0f8612dfc5f3247f5dbfdaf4389d2b0e55461a57d40a497444abb503"}
Sep 17 17:50:39.926884 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.926877    2562 scope.go:117] "RemoveContainer" containerID="34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a"
Sep 17 17:50:39.935198 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.935181    2562 scope.go:117] "RemoveContainer" containerID="081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b"
Sep 17 17:50:39.942697 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.942678    2562 scope.go:117] "RemoveContainer" containerID="589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219"
Sep 17 17:50:39.950014 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.949995    2562 scope.go:117] "RemoveContainer" containerID="eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a"
Sep 17 17:50:39.958134 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.958108    2562 scope.go:117] "RemoveContainer" containerID="34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a"
Sep 17 17:50:39.958453 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:50:39.958435    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a\": container with ID starting with 34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a not found: ID does not exist" containerID="34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a"
Sep 17 17:50:39.958517 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.958463    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a"} err="failed to get container status \"34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a\": rpc error: code = NotFound desc = could not find container \"34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a\": container with ID starting with 34863f46d87f74d7be178b131ae41a30a9f6b3e9eb152bf1aaefc04d219da83a not found: ID does not exist"
Sep 17 17:50:39.958517 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.958482    2562 scope.go:117] "RemoveContainer" containerID="081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b"
Sep 17 17:50:39.958754 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:50:39.958734    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b\": container with ID starting with 081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b not found: ID does not exist" containerID="081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b"
Sep 17 17:50:39.958793 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.958761    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b"} err="failed to get container status \"081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b\": rpc error: code = NotFound desc = could not find container \"081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b\": container with ID starting with 081ebbab78a79a9fed5f4c24a26098979347305f227351a2707b4bd057c9c89b not found: ID does not exist"
Sep 17 17:50:39.958793 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.958776    2562 scope.go:117] "RemoveContainer" containerID="589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219"
Sep 17 17:50:39.959047 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:50:39.959026    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219\": container with ID starting with 589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219 not found: ID does not exist" containerID="589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219"
Sep 17 17:50:39.959156 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.959049    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219"} err="failed to get container status \"589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219\": rpc error: code = NotFound desc = could not find container \"589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219\": container with ID starting with 589480f3abb3267ceee549d8bbbf5731330c677d12f2cc8f13e7dd9d38a05219 not found: ID does not exist"
Sep 17 17:50:39.959156 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.959065    2562 scope.go:117] "RemoveContainer" containerID="eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a"
Sep 17 17:50:39.959651 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:50:39.959312    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a\": container with ID starting with eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a not found: ID does not exist" containerID="eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a"
Sep 17 17:50:39.959651 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.959352    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a"} err="failed to get container status \"eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a\": rpc error: code = NotFound desc = could not find container \"eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a\": container with ID starting with eb4c1059e5d51ba64f08c8e02256a4ce5911c86f655a700e3f37961ba8d3412a not found: ID does not exist"
Sep 17 17:50:39.959651 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.959599    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"]
Sep 17 17:50:39.965733 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:39.965708    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-745858bf58-kjh4m"]
Sep 17 17:50:41.727343 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:41.727298    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" path="/var/lib/kubelet/pods/8eb4058a-4006-48a4-9322-c5e553f94532/volumes"
Sep 17 17:50:42.870246 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:42.870204    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:50:42.870659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:42.870616    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:52.870426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:52.870389    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:50:52.870905 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:50:52.870880    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:02.870939 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:02.870896    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:51:02.871467 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:02.871438    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:12.870187 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:12.870144    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:51:12.870765 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:12.870694    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:22.870190 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:22.870146    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:51:22.870646 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:22.870607    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:32.871011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:32.870964    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:51:32.871482 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:32.871413    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:42.871539 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:42.871508    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:51:42.871959 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:42.871753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:51:54.485532 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:54.485496    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"]
Sep 17 17:51:54.486022 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:54.485820    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" containerID="cri-o://26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85" gracePeriod=30
Sep 17 17:51:54.486022 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:54.485858    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" containerID="cri-o://e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a" gracePeriod=30
Sep 17 17:51:54.486130 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:54.486088    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" containerID="cri-o://52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d" gracePeriod=30
Sep 17 17:51:55.282840 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:55.282808    2562 generic.go:356] "Generic (PLEG): container finished" podID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerID="e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a" exitCode=2
Sep 17 17:51:55.283019 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:55.282861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerDied","Data":"e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a"}
Sep 17 17:51:57.865766 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:57.865725    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 17:51:59.305009 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:59.304893    2562 generic.go:356] "Generic (PLEG): container finished" podID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerID="26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85" exitCode=0
Sep 17 17:51:59.305009 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:51:59.304947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerDied","Data":"26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85"}
Sep 17 17:52:02.865805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:02.865766    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 17:52:02.870579 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:02.870550    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:52:02.870894 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:02.870874    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:07.866553 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:07.866510    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 17:52:07.866946 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:07.866637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:52:12.866102 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:12.866058    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 17:52:12.870476 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:12.870447    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:52:12.870906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:12.870884    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:17.866643 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:17.866599    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 17:52:21.277963 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.277926    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"]
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278269    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="storage-initializer"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278279    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="storage-initializer"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278290    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278296    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278304    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278310    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278318    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container"
Sep 17 17:52:21.278322 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278324    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container"
Sep 17 17:52:21.278595 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278408    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kube-rbac-proxy"
Sep 17 17:52:21.278595 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278416    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="agent"
Sep 17 17:52:21.278595 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.278425    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb4058a-4006-48a4-9322-c5e553f94532" containerName="kserve-container"
Sep 17 17:52:21.281496 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.281478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.286580 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.286558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 17:52:21.287532 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.287514    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:52:21.309535 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.309508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssc5\" (UniqueName: \"kubernetes.io/projected/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-api-access-tssc5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.309677 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.309545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab8bb0a-38ab-402b-be7a-1420bf97a245-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.309677 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.309576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.309677 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.309662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.309780 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.309682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.316250 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.316219    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"]
Sep 17 17:52:21.410051 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410051 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410315 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssc5\" (UniqueName: \"kubernetes.io/projected/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-api-access-tssc5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410315 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab8bb0a-38ab-402b-be7a-1420bf97a245-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410315 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:52:21.410198    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:52:21.410315 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410315 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:52:21.410276    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls podName:dab8bb0a-38ab-402b-be7a-1420bf97a245 nodeName:}" failed. No retries permitted until 2026-09-17 17:52:21.9102544 +0000 UTC m=+1086.358084848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" (UID: "dab8bb0a-38ab-402b-be7a-1420bf97a245") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:52:21.410550 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab8bb0a-38ab-402b-be7a-1420bf97a245-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.410845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.410837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.423029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.423004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssc5\" (UniqueName: \"kubernetes.io/projected/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-api-access-tssc5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.913413 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.913382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:21.915692 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:21.915663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:22.193376 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.193261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:22.468461 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.468437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"]
Sep 17 17:52:22.470454 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:52:22.470428    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab8bb0a_38ab_402b_be7a_1420bf97a245.slice/crio-0c248e8c9e09e429e26a232d43e149bd6f3e482241bc3fdb6d8db0a0456f0e25 WatchSource:0}: Error finding container 0c248e8c9e09e429e26a232d43e149bd6f3e482241bc3fdb6d8db0a0456f0e25: Status 404 returned error can't find the container with id 0c248e8c9e09e429e26a232d43e149bd6f3e482241bc3fdb6d8db0a0456f0e25
Sep 17 17:52:22.866476 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.866435    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 17:52:22.870843 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.870807    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 17:52:22.870973 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.870933    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:52:22.871151 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.871129    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:22.871221 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:22.871210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:52:23.419126 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:23.419095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerStarted","Data":"2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7"}
Sep 17 17:52:23.419324 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:23.419193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerStarted","Data":"0c248e8c9e09e429e26a232d43e149bd6f3e482241bc3fdb6d8db0a0456f0e25"}
Sep 17 17:52:24.671375 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.671352    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:52:24.741658 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.741591    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-cabundle-cert\") pod \"a04de597-9614-4eea-91db-1bfc009c0ab0\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") "
Sep 17 17:52:24.741658 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.741625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a04de597-9614-4eea-91db-1bfc009c0ab0-kserve-provision-location\") pod \"a04de597-9614-4eea-91db-1bfc009c0ab0\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") "
Sep 17 17:52:24.741834 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.741685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-rbac-proxy-sar-config\") pod \"a04de597-9614-4eea-91db-1bfc009c0ab0\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") "
Sep 17 17:52:24.741834 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.741700    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tjjz\" (UniqueName: \"kubernetes.io/projected/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-api-access-9tjjz\") pod \"a04de597-9614-4eea-91db-1bfc009c0ab0\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") "
Sep 17 17:52:24.741834 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.741721    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls\") pod \"a04de597-9614-4eea-91db-1bfc009c0ab0\" (UID: \"a04de597-9614-4eea-91db-1bfc009c0ab0\") "
Sep 17 17:52:24.742014 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.741986    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a04de597-9614-4eea-91db-1bfc009c0ab0" (UID: "a04de597-9614-4eea-91db-1bfc009c0ab0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:24.742148 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.742019    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04de597-9614-4eea-91db-1bfc009c0ab0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a04de597-9614-4eea-91db-1bfc009c0ab0" (UID: "a04de597-9614-4eea-91db-1bfc009c0ab0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:24.742148 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.742067    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a04de597-9614-4eea-91db-1bfc009c0ab0" (UID: "a04de597-9614-4eea-91db-1bfc009c0ab0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:24.743897 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.743871    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-api-access-9tjjz" (OuterVolumeSpecName: "kube-api-access-9tjjz") pod "a04de597-9614-4eea-91db-1bfc009c0ab0" (UID: "a04de597-9614-4eea-91db-1bfc009c0ab0"). InnerVolumeSpecName "kube-api-access-9tjjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:24.744012 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.743903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a04de597-9614-4eea-91db-1bfc009c0ab0" (UID: "a04de597-9614-4eea-91db-1bfc009c0ab0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:24.843047 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.843015    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a04de597-9614-4eea-91db-1bfc009c0ab0-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:52:24.843047 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.843042    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:52:24.843047 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.843052    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjjz\" (UniqueName: \"kubernetes.io/projected/a04de597-9614-4eea-91db-1bfc009c0ab0-kube-api-access-9tjjz\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:52:24.843269 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.843062    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a04de597-9614-4eea-91db-1bfc009c0ab0-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:52:24.843269 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:24.843072    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a04de597-9614-4eea-91db-1bfc009c0ab0-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:52:25.430743 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.430708    2562 generic.go:356] "Generic (PLEG): container finished" podID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerID="52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d" exitCode=0
Sep 17 17:52:25.430941 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.430765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerDied","Data":"52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d"}
Sep 17 17:52:25.430941 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.430836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99" event={"ID":"a04de597-9614-4eea-91db-1bfc009c0ab0","Type":"ContainerDied","Data":"fa0083163fbffd791fe264283a257ff58f3bc89f9b1b3470999eae9847ab5995"}
Sep 17 17:52:25.430941 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.430767    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"
Sep 17 17:52:25.430941 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.430878    2562 scope.go:117] "RemoveContainer" containerID="52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d"
Sep 17 17:52:25.439449 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.439432    2562 scope.go:117] "RemoveContainer" containerID="e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a"
Sep 17 17:52:25.446804 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.446787    2562 scope.go:117] "RemoveContainer" containerID="26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85"
Sep 17 17:52:25.453874 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.453851    2562 scope.go:117] "RemoveContainer" containerID="11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af"
Sep 17 17:52:25.461023 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461007    2562 scope.go:117] "RemoveContainer" containerID="52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d"
Sep 17 17:52:25.461282 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:52:25.461265    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d\": container with ID starting with 52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d not found: ID does not exist" containerID="52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d"
Sep 17 17:52:25.461354 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461290    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d"} err="failed to get container status \"52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d\": rpc error: code = NotFound desc = could not find container \"52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d\": container with ID starting with 52a395a5f3f2c9c096d891e27f434056499bf5269a15e3bbb6be53bbb9640d9d not found: ID does not exist"
Sep 17 17:52:25.461354 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461307    2562 scope.go:117] "RemoveContainer" containerID="e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a"
Sep 17 17:52:25.461573 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:52:25.461557    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a\": container with ID starting with e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a not found: ID does not exist" containerID="e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a"
Sep 17 17:52:25.461623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461578    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a"} err="failed to get container status \"e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a\": rpc error: code = NotFound desc = could not find container \"e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a\": container with ID starting with e7a7809ae25ba50f4595a36f63a4cd600e57ce35c622acf7af3632cd0a454d2a not found: ID does not exist"
Sep 17 17:52:25.461623 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461592    2562 scope.go:117] "RemoveContainer" containerID="26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85"
Sep 17 17:52:25.461884 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:52:25.461860    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85\": container with ID starting with 26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85 not found: ID does not exist" containerID="26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85"
Sep 17 17:52:25.461958 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461888    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85"} err="failed to get container status \"26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85\": rpc error: code = NotFound desc = could not find container \"26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85\": container with ID starting with 26016fa0287e8c04d037bcb48759c9d92b9f928db10d4554216795a647617c85 not found: ID does not exist"
Sep 17 17:52:25.461958 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.461910    2562 scope.go:117] "RemoveContainer" containerID="11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af"
Sep 17 17:52:25.462198 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:52:25.462176    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af\": container with ID starting with 11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af not found: ID does not exist" containerID="11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af"
Sep 17 17:52:25.462260 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.462202    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af"} err="failed to get container status \"11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af\": rpc error: code = NotFound desc = could not find container \"11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af\": container with ID starting with 11f3ea303a9607b2d9ffc1f2cf7306866a1fccdb265bd7d6c805e6eb3c0bd9af not found: ID does not exist"
Sep 17 17:52:25.464898 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.464877    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"]
Sep 17 17:52:25.472823 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.472804    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6bdfd88f5f-sqg99"]
Sep 17 17:52:25.726772 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:25.726700    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" path="/var/lib/kubelet/pods/a04de597-9614-4eea-91db-1bfc009c0ab0/volumes"
Sep 17 17:52:26.437948 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:26.437917    2562 generic.go:356] "Generic (PLEG): container finished" podID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerID="2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7" exitCode=0
Sep 17 17:52:26.438119 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:26.437993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerDied","Data":"2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7"}
Sep 17 17:52:26.439367 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:26.439352    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:52:51.588741 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:51.588704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerStarted","Data":"fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98"}
Sep 17 17:52:51.589128 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:51.588769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerStarted","Data":"9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92"}
Sep 17 17:52:51.589128 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:51.589009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:51.613940 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:51.613893    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podStartSLOduration=2.683903936 podStartE2EDuration="30.613878932s" totalImagesPullingTime="24.170121047s" totalInitContainerRuntime="3.759853949s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:51.61122821 +0000 UTC m=+1116.059058709" watchObservedRunningTime="2026-09-17 17:52:51.613878932 +0000 UTC m=+1116.061709457"
Sep 17 17:52:52.599148 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:52.599107    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:52:58.609713 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:52:58.609681    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:53:28.610770 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:53:28.610729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:53:38.611200 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:53:38.611122    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:53:48.610406 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:53:48.610365    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:53:58.610634 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:53:58.610596    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 17:54:07.320947 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:07.320913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:54:11.527681 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:11.527649    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"]
Sep 17 17:54:11.528255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:11.527904    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container" containerID="cri-o://9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92" gracePeriod=30
Sep 17 17:54:11.528255 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:11.527975    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kube-rbac-proxy" containerID="cri-o://fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98" gracePeriod=30
Sep 17 17:54:11.951249 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:11.951220    2562 generic.go:356] "Generic (PLEG): container finished" podID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerID="fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98" exitCode=2
Sep 17 17:54:11.951429 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:11.951288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerDied","Data":"fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98"}
Sep 17 17:54:13.605282 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:13.605241    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 17:54:16.269140 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.269114    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:54:16.375385 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tssc5\" (UniqueName: \"kubernetes.io/projected/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-api-access-tssc5\") pod \"dab8bb0a-38ab-402b-be7a-1420bf97a245\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") "
Sep 17 17:54:16.375523 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls\") pod \"dab8bb0a-38ab-402b-be7a-1420bf97a245\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") "
Sep 17 17:54:16.375523 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375438    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab8bb0a-38ab-402b-be7a-1420bf97a245-kserve-provision-location\") pod \"dab8bb0a-38ab-402b-be7a-1420bf97a245\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") "
Sep 17 17:54:16.375523 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-rbac-proxy-sar-config\") pod \"dab8bb0a-38ab-402b-be7a-1420bf97a245\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") "
Sep 17 17:54:16.375690 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-cabundle-cert\") pod \"dab8bb0a-38ab-402b-be7a-1420bf97a245\" (UID: \"dab8bb0a-38ab-402b-be7a-1420bf97a245\") "
Sep 17 17:54:16.375844 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab8bb0a-38ab-402b-be7a-1420bf97a245-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dab8bb0a-38ab-402b-be7a-1420bf97a245" (UID: "dab8bb0a-38ab-402b-be7a-1420bf97a245"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:16.375967 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dab8bb0a-38ab-402b-be7a-1420bf97a245" (UID: "dab8bb0a-38ab-402b-be7a-1420bf97a245"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:16.375967 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.375912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dab8bb0a-38ab-402b-be7a-1420bf97a245" (UID: "dab8bb0a-38ab-402b-be7a-1420bf97a245"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:16.377368 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.377342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dab8bb0a-38ab-402b-be7a-1420bf97a245" (UID: "dab8bb0a-38ab-402b-be7a-1420bf97a245"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:16.377368 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.377352    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-api-access-tssc5" (OuterVolumeSpecName: "kube-api-access-tssc5") pod "dab8bb0a-38ab-402b-be7a-1420bf97a245" (UID: "dab8bb0a-38ab-402b-be7a-1420bf97a245"). InnerVolumeSpecName "kube-api-access-tssc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:16.476915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.476893    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:54:16.476915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.476913    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab8bb0a-38ab-402b-be7a-1420bf97a245-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:54:16.477038 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.476922    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tssc5\" (UniqueName: \"kubernetes.io/projected/dab8bb0a-38ab-402b-be7a-1420bf97a245-kube-api-access-tssc5\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:54:16.477038 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.476931    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab8bb0a-38ab-402b-be7a-1420bf97a245-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:54:16.477038 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.476940    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab8bb0a-38ab-402b-be7a-1420bf97a245-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:54:16.768081 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768007    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"]
Sep 17 17:54:16.768360 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768347    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768361    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768374    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="storage-initializer"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768379    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="storage-initializer"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768388    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768394    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768406    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container"
Sep 17 17:54:16.768412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768413    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768424    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kube-rbac-proxy"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768429    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kube-rbac-proxy"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768437    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768442    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768449    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="storage-initializer"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768454    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="storage-initializer"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768503    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="agent"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768511    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kserve-container"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768517    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerName="kube-rbac-proxy"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768524    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kserve-container"
Sep 17 17:54:16.768659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.768528    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04de597-9614-4eea-91db-1bfc009c0ab0" containerName="kube-rbac-proxy"
Sep 17 17:54:16.771709 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.771692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.777779 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.777759    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:54:16.777898 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.777822    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 17:54:16.793024 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.793002    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"]
Sep 17 17:54:16.879615 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.879581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.879757 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.879673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.879757 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.879712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.879757 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.879739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.879870 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.879760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqbt\" (UniqueName: \"kubernetes.io/projected/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-api-access-hsqbt\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.975988 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.975957    2562 generic.go:356] "Generic (PLEG): container finished" podID="dab8bb0a-38ab-402b-be7a-1420bf97a245" containerID="9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92" exitCode=0
Sep 17 17:54:16.976127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.975990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerDied","Data":"9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92"}
Sep 17 17:54:16.976127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.976026    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"
Sep 17 17:54:16.976127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.976067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47" event={"ID":"dab8bb0a-38ab-402b-be7a-1420bf97a245","Type":"ContainerDied","Data":"0c248e8c9e09e429e26a232d43e149bd6f3e482241bc3fdb6d8db0a0456f0e25"}
Sep 17 17:54:16.976127 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.976102    2562 scope.go:117] "RemoveContainer" containerID="fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98"
Sep 17 17:54:16.980341 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.980309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.980468 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.980451    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.980545 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.980507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.980545 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.980536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqbt\" (UniqueName: \"kubernetes.io/projected/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-api-access-hsqbt\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.980651 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.980619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.980840 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.980815    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.981111 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.981082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.981201 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.981162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.982999 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.982979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.985653 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.985638    2562 scope.go:117] "RemoveContainer" containerID="9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92"
Sep 17 17:54:16.992258 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.992233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqbt\" (UniqueName: \"kubernetes.io/projected/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-api-access-hsqbt\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:16.993338 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:16.993264    2562 scope.go:117] "RemoveContainer" containerID="2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7"
Sep 17 17:54:17.000184 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.000170    2562 scope.go:117] "RemoveContainer" containerID="fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98"
Sep 17 17:54:17.000532 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:54:17.000515    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98\": container with ID starting with fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98 not found: ID does not exist" containerID="fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98"
Sep 17 17:54:17.000605 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.000538    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98"} err="failed to get container status \"fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98\": rpc error: code = NotFound desc = could not find container \"fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98\": container with ID starting with fa74264ed9cf950ffc3309e4af0b2e52c2fc162c1c97da6d4e968d580d638a98 not found: ID does not exist"
Sep 17 17:54:17.000605 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.000553    2562 scope.go:117] "RemoveContainer" containerID="9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92"
Sep 17 17:54:17.000804 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:54:17.000784    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92\": container with ID starting with 9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92 not found: ID does not exist" containerID="9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92"
Sep 17 17:54:17.000845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.000814    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92"} err="failed to get container status \"9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92\": rpc error: code = NotFound desc = could not find container \"9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92\": container with ID starting with 9fabe6c328ae1b4f4a82f9f4a939829852740d86de513376b0223fc909903c92 not found: ID does not exist"
Sep 17 17:54:17.000845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.000829    2562 scope.go:117] "RemoveContainer" containerID="2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7"
Sep 17 17:54:17.001070 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:54:17.001056    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7\": container with ID starting with 2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7 not found: ID does not exist" containerID="2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7"
Sep 17 17:54:17.001137 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.001073    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7"} err="failed to get container status \"2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7\": rpc error: code = NotFound desc = could not find container \"2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7\": container with ID starting with 2d937b0cf686f3539d277e01c51e48bf906ddae6d4588cfe4dca4bcc4bd9a8f7 not found: ID does not exist"
Sep 17 17:54:17.006584 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.006564    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"]
Sep 17 17:54:17.010601 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.010579    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-bmq47"]
Sep 17 17:54:17.081727 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.081702    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:17.367540 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.367380    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"]
Sep 17 17:54:17.369900 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:54:17.369877    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcaa38e3_2a11_4cb7_a3cf_a39552e2080c.slice/crio-3d6be150b03666b9e5e285b2023cd31d388ae25b49cc1880f641186f092f1d79 WatchSource:0}: Error finding container 3d6be150b03666b9e5e285b2023cd31d388ae25b49cc1880f641186f092f1d79: Status 404 returned error can't find the container with id 3d6be150b03666b9e5e285b2023cd31d388ae25b49cc1880f641186f092f1d79
Sep 17 17:54:17.716921 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.716832    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:54:17.720149 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.720129    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:54:17.728439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.728415    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab8bb0a-38ab-402b-be7a-1420bf97a245" path="/var/lib/kubelet/pods/dab8bb0a-38ab-402b-be7a-1420bf97a245/volumes"
Sep 17 17:54:17.982466 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.982364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerStarted","Data":"0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719"}
Sep 17 17:54:17.982466 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:17.982436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerStarted","Data":"3d6be150b03666b9e5e285b2023cd31d388ae25b49cc1880f641186f092f1d79"}
Sep 17 17:54:22.002490 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:22.002455    2562 generic.go:356] "Generic (PLEG): container finished" podID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerID="0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719" exitCode=0
Sep 17 17:54:22.002937 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:22.002533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerDied","Data":"0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719"}
Sep 17 17:54:23.008672 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:23.008641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerStarted","Data":"ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b"}
Sep 17 17:54:23.009039 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:23.008703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerStarted","Data":"3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291"}
Sep 17 17:54:23.009039 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:23.008911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:23.009111 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:23.009067    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:23.036742 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:23.036700    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podStartSLOduration=3.391869543 podStartE2EDuration="7.036682864s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.644813321s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:23.036444058 +0000 UTC m=+1207.484274539" watchObservedRunningTime="2026-09-17 17:54:23.036682864 +0000 UTC m=+1207.484513318"
Sep 17 17:54:29.019218 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:29.019185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:54:59.020145 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:54:59.020098    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:09.020115 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:09.020067    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:19.019620 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:19.019581    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:29.020159 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:29.020115    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:39.526780 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:39.526749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:55:46.732672 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:46.732639    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"]
Sep 17 17:55:46.733122 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:46.732894    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container" containerID="cri-o://3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291" gracePeriod=30
Sep 17 17:55:46.733122 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:46.732933    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kube-rbac-proxy" containerID="cri-o://ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b" gracePeriod=30
Sep 17 17:55:47.396515 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:47.396485    2562 generic.go:356] "Generic (PLEG): container finished" podID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerID="ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b" exitCode=2
Sep 17 17:55:47.396683 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:47.396559    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerDied","Data":"ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b"}
Sep 17 17:55:49.014049 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:49.014005    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 17:55:49.020111 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:49.020081    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.33:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 17:55:51.176972 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.176949    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:55:51.332398 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsqbt\" (UniqueName: \"kubernetes.io/projected/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-api-access-hsqbt\") pod \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") "
Sep 17 17:55:51.332554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-proxy-tls\") pod \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") "
Sep 17 17:55:51.332554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332459    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-cabundle-cert\") pod \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") "
Sep 17 17:55:51.332554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332475    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-rbac-proxy-sar-config\") pod \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") "
Sep 17 17:55:51.332554 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332506    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kserve-provision-location\") pod \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\" (UID: \"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c\") "
Sep 17 17:55:51.332879 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332846    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" (UID: "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:51.332879 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332859    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" (UID: "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:51.332879 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.332847    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" (UID: "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:51.334423 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.334399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-api-access-hsqbt" (OuterVolumeSpecName: "kube-api-access-hsqbt") pod "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" (UID: "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c"). InnerVolumeSpecName "kube-api-access-hsqbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:51.334568 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.334545    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" (UID: "fcaa38e3-2a11-4cb7-a3cf-a39552e2080c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:51.416931 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.416895    2562 generic.go:356] "Generic (PLEG): container finished" podID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerID="3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291" exitCode=0
Sep 17 17:55:51.417077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.416947    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"
Sep 17 17:55:51.417077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.416970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerDied","Data":"3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291"}
Sep 17 17:55:51.417077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.417017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899" event={"ID":"fcaa38e3-2a11-4cb7-a3cf-a39552e2080c","Type":"ContainerDied","Data":"3d6be150b03666b9e5e285b2023cd31d388ae25b49cc1880f641186f092f1d79"}
Sep 17 17:55:51.417077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.417042    2562 scope.go:117] "RemoveContainer" containerID="ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b"
Sep 17 17:55:51.425754 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.425737    2562 scope.go:117] "RemoveContainer" containerID="3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291"
Sep 17 17:55:51.432750 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.432732    2562 scope.go:117] "RemoveContainer" containerID="0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719"
Sep 17 17:55:51.433019 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.433001    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:55:51.433067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.433022    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqbt\" (UniqueName: \"kubernetes.io/projected/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-api-access-hsqbt\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:55:51.433067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.433036    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:55:51.433067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.433050    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:55:51.433067 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.433058    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:55:51.439905 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.439890    2562 scope.go:117] "RemoveContainer" containerID="ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b"
Sep 17 17:55:51.440181 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:55:51.440154    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b\": container with ID starting with ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b not found: ID does not exist" containerID="ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b"
Sep 17 17:55:51.440225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.440189    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b"} err="failed to get container status \"ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b\": rpc error: code = NotFound desc = could not find container \"ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b\": container with ID starting with ab785c0603c96e08e8871612c569fbd9cdf7c55babb61201a24a239606ccee6b not found: ID does not exist"
Sep 17 17:55:51.440225 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.440209    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"]
Sep 17 17:55:51.440296 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.440212    2562 scope.go:117] "RemoveContainer" containerID="3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291"
Sep 17 17:55:51.440504 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:55:51.440474    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291\": container with ID starting with 3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291 not found: ID does not exist" containerID="3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291"
Sep 17 17:55:51.440504 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.440499    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291"} err="failed to get container status \"3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291\": rpc error: code = NotFound desc = could not find container \"3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291\": container with ID starting with 3ce80aa8cc56f28620d9b65448bf870b85265ba9e43204a0de62cdfde2e89291 not found: ID does not exist"
Sep 17 17:55:51.440659 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.440514    2562 scope.go:117] "RemoveContainer" containerID="0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719"
Sep 17 17:55:51.440752 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:55:51.440736    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719\": container with ID starting with 0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719 not found: ID does not exist" containerID="0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719"
Sep 17 17:55:51.440802 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.440760    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719"} err="failed to get container status \"0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719\": rpc error: code = NotFound desc = could not find container \"0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719\": container with ID starting with 0e026a9e9e3be969b9a1b172f4052d8763478f5105c5c4dff397ab357cb46719 not found: ID does not exist"
Sep 17 17:55:51.444532 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.444511    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gl899"]
Sep 17 17:55:51.732157 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.732080    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" path="/var/lib/kubelet/pods/fcaa38e3-2a11-4cb7-a3cf-a39552e2080c/volumes"
Sep 17 17:55:51.918463 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918429    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"]
Sep 17 17:55:51.918790 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918777    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="storage-initializer"
Sep 17 17:55:51.918830 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918792    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="storage-initializer"
Sep 17 17:55:51.918830 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918804    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kube-rbac-proxy"
Sep 17 17:55:51.918830 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918810    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kube-rbac-proxy"
Sep 17 17:55:51.918830 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918828    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container"
Sep 17 17:55:51.918950 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918835    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container"
Sep 17 17:55:51.918950 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918887    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kserve-container"
Sep 17 17:55:51.918950 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.918896    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcaa38e3-2a11-4cb7-a3cf-a39552e2080c" containerName="kube-rbac-proxy"
Sep 17 17:55:51.923307 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.923289    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:51.925915 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.925894    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:55:51.926063 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.926038    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:55:51.926161 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.925966    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:55:51.926161 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.926042    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:55:51.926274 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.925893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 17:55:51.926274 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.926043    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:55:51.926274 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.926047    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:55:51.933387 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:51.933365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"]
Sep 17 17:55:52.039216 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.039143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26b49896-414a-4c72-ae8d-e3efd07b8dce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.039216 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.039195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.039216 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.039213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.039454 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.039278    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6w2c\" (UniqueName: \"kubernetes.io/projected/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-api-access-g6w2c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.039454 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.039339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.140607 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.140574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26b49896-414a-4c72-ae8d-e3efd07b8dce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.140760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.140614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.140760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.140633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.140760 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.140655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6w2c\" (UniqueName: \"kubernetes.io/projected/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-api-access-g6w2c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.140925 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.140780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.140979 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:55:52.140961    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 17:55:52.141027 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.140976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26b49896-414a-4c72-ae8d-e3efd07b8dce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.141083 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:55:52.141035    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls podName:26b49896-414a-4c72-ae8d-e3efd07b8dce nodeName:}" failed. No retries permitted until 2026-09-17 17:55:52.641014489 +0000 UTC m=+1297.088844936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" (UID: "26b49896-414a-4c72-ae8d-e3efd07b8dce") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 17:55:52.141341 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.141308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.141412 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.141309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.151984 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.151966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6w2c\" (UniqueName: \"kubernetes.io/projected/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-api-access-g6w2c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.645677 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.645633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.647900 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.647881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:52.833904 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:52.833875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:53.111658 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:53.111633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"]
Sep 17 17:55:53.114203 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:55:53.114179    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b49896_414a_4c72_ae8d_e3efd07b8dce.slice/crio-055f79b0339f1ab5b61541f816a4392a1618eacf41b82d00aa12fce073a23042 WatchSource:0}: Error finding container 055f79b0339f1ab5b61541f816a4392a1618eacf41b82d00aa12fce073a23042: Status 404 returned error can't find the container with id 055f79b0339f1ab5b61541f816a4392a1618eacf41b82d00aa12fce073a23042
Sep 17 17:55:53.429394 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:53.429302    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerStarted","Data":"06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628"}
Sep 17 17:55:53.429394 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:53.429390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerStarted","Data":"055f79b0339f1ab5b61541f816a4392a1618eacf41b82d00aa12fce073a23042"}
Sep 17 17:55:57.448522 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:57.448487    2562 generic.go:356] "Generic (PLEG): container finished" podID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerID="06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628" exitCode=0
Sep 17 17:55:57.448866 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:57.448542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerDied","Data":"06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628"}
Sep 17 17:55:58.455713 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:58.455679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerStarted","Data":"ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6"}
Sep 17 17:55:58.456101 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:58.455769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerStarted","Data":"9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8"}
Sep 17 17:55:58.456101 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:58.455971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:55:58.479527 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:58.479483    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podStartSLOduration=3.426768855 podStartE2EDuration="7.479470203s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.052701348s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:58.478684768 +0000 UTC m=+1302.926515269" watchObservedRunningTime="2026-09-17 17:55:58.479470203 +0000 UTC m=+1302.927300656"
Sep 17 17:55:59.461102 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:55:59.461074    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:56:05.471874 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:56:05.471848    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:56:35.473592 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:56:35.473514    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 17:56:45.472521 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:56:45.472481    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 17:56:55.473309 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:56:55.473269    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 17:57:05.472947 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:05.472907    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 17:57:16.018486 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:16.018442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:57:22.013626 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:22.013590    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"]
Sep 17 17:57:22.014073 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:22.013905    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container" containerID="cri-o://9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8" gracePeriod=30
Sep 17 17:57:22.014073 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:22.014016    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kube-rbac-proxy" containerID="cri-o://ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6" gracePeriod=30
Sep 17 17:57:22.835155 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:22.835124    2562 generic.go:356] "Generic (PLEG): container finished" podID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerID="ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6" exitCode=2
Sep 17 17:57:22.835308 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:22.835195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerDied","Data":"ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6"}
Sep 17 17:57:25.466152 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:25.466111    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 17:57:25.472486 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:25.472460    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 17:57:26.453158 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.453130    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:57:26.469922 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.469898    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-rbac-proxy-sar-config\") pod \"26b49896-414a-4c72-ae8d-e3efd07b8dce\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") "
Sep 17 17:57:26.470248 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.469930    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26b49896-414a-4c72-ae8d-e3efd07b8dce-kserve-provision-location\") pod \"26b49896-414a-4c72-ae8d-e3efd07b8dce\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") "
Sep 17 17:57:26.470248 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.469976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-cabundle-cert\") pod \"26b49896-414a-4c72-ae8d-e3efd07b8dce\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") "
Sep 17 17:57:26.470248 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.470001    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls\") pod \"26b49896-414a-4c72-ae8d-e3efd07b8dce\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") "
Sep 17 17:57:26.470248 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.470018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6w2c\" (UniqueName: \"kubernetes.io/projected/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-api-access-g6w2c\") pod \"26b49896-414a-4c72-ae8d-e3efd07b8dce\" (UID: \"26b49896-414a-4c72-ae8d-e3efd07b8dce\") "
Sep 17 17:57:26.470471 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.470318    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26b49896-414a-4c72-ae8d-e3efd07b8dce" (UID: "26b49896-414a-4c72-ae8d-e3efd07b8dce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:26.470471 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.470351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26b49896-414a-4c72-ae8d-e3efd07b8dce" (UID: "26b49896-414a-4c72-ae8d-e3efd07b8dce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:26.470471 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.470363    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b49896-414a-4c72-ae8d-e3efd07b8dce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26b49896-414a-4c72-ae8d-e3efd07b8dce" (UID: "26b49896-414a-4c72-ae8d-e3efd07b8dce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:26.472058 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.472032    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26b49896-414a-4c72-ae8d-e3efd07b8dce" (UID: "26b49896-414a-4c72-ae8d-e3efd07b8dce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:26.472151 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.472057    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-api-access-g6w2c" (OuterVolumeSpecName: "kube-api-access-g6w2c") pod "26b49896-414a-4c72-ae8d-e3efd07b8dce" (UID: "26b49896-414a-4c72-ae8d-e3efd07b8dce"). InnerVolumeSpecName "kube-api-access-g6w2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:26.570702 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.570677    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26b49896-414a-4c72-ae8d-e3efd07b8dce-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.570702 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.570698    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6w2c\" (UniqueName: \"kubernetes.io/projected/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-api-access-g6w2c\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.570841 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.570708    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.570841 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.570716    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26b49896-414a-4c72-ae8d-e3efd07b8dce-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.570841 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.570725    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26b49896-414a-4c72-ae8d-e3efd07b8dce-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:26.854859 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.854831    2562 generic.go:356] "Generic (PLEG): container finished" podID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerID="9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8" exitCode=0
Sep 17 17:57:26.854976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.854897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"
Sep 17 17:57:26.854976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.854896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerDied","Data":"9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8"}
Sep 17 17:57:26.855063 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.855019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24" event={"ID":"26b49896-414a-4c72-ae8d-e3efd07b8dce","Type":"ContainerDied","Data":"055f79b0339f1ab5b61541f816a4392a1618eacf41b82d00aa12fce073a23042"}
Sep 17 17:57:26.855063 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.855057    2562 scope.go:117] "RemoveContainer" containerID="ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6"
Sep 17 17:57:26.863500 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.863482    2562 scope.go:117] "RemoveContainer" containerID="9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8"
Sep 17 17:57:26.870728 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.870712    2562 scope.go:117] "RemoveContainer" containerID="06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628"
Sep 17 17:57:26.877676 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.877656    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"]
Sep 17 17:57:26.878079 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.878069    2562 scope.go:117] "RemoveContainer" containerID="ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6"
Sep 17 17:57:26.878301 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:57:26.878282    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6\": container with ID starting with ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6 not found: ID does not exist" containerID="ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6"
Sep 17 17:57:26.878403 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.878305    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6"} err="failed to get container status \"ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6\": rpc error: code = NotFound desc = could not find container \"ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6\": container with ID starting with ccf19f74def482715a45042f89b624af61b06a74be2618546dccb86b90fc21f6 not found: ID does not exist"
Sep 17 17:57:26.878403 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.878320    2562 scope.go:117] "RemoveContainer" containerID="9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8"
Sep 17 17:57:26.878565 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:57:26.878547    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8\": container with ID starting with 9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8 not found: ID does not exist" containerID="9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8"
Sep 17 17:57:26.878611 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.878573    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8"} err="failed to get container status \"9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8\": rpc error: code = NotFound desc = could not find container \"9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8\": container with ID starting with 9b10b2888ba3fbab4c2d7e040c7fc16818e9f4e153bd5d671a3aa98da2d7a2b8 not found: ID does not exist"
Sep 17 17:57:26.878611 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.878588    2562 scope.go:117] "RemoveContainer" containerID="06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628"
Sep 17 17:57:26.878821 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:57:26.878806    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628\": container with ID starting with 06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628 not found: ID does not exist" containerID="06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628"
Sep 17 17:57:26.878874 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.878823    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628"} err="failed to get container status \"06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628\": rpc error: code = NotFound desc = could not find container \"06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628\": container with ID starting with 06527157f651d716d8247087b0886178c691b99baf1daf17951f553c37ca9628 not found: ID does not exist"
Sep 17 17:57:26.883540 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:26.883521    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6gp24"]
Sep 17 17:57:27.727268 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:27.727236    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" path="/var/lib/kubelet/pods/26b49896-414a-4c72-ae8d-e3efd07b8dce/volumes"
Sep 17 17:57:44.744426 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744394    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-cc7ec9a4"]
Sep 17 17:57:44.744769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744692    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="storage-initializer"
Sep 17 17:57:44.744769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744703    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="storage-initializer"
Sep 17 17:57:44.744769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744720    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container"
Sep 17 17:57:44.744769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744725    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container"
Sep 17 17:57:44.744769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744741    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kube-rbac-proxy"
Sep 17 17:57:44.744769 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744748    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kube-rbac-proxy"
Sep 17 17:57:44.744953 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744790    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kube-rbac-proxy"
Sep 17 17:57:44.744953 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.744799    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b49896-414a-4c72-ae8d-e3efd07b8dce" containerName="kserve-container"
Sep 17 17:57:44.747602 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.747586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:44.756125 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.756104    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-cc7ec9a4"]
Sep 17 17:57:44.806103 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.806079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdds\" (UniqueName: \"kubernetes.io/projected/fbdc3277-8613-43df-b73c-18c2abf047d3-kube-api-access-9bdds\") pod \"curl-metrics-cc7ec9a4\" (UID: \"fbdc3277-8613-43df-b73c-18c2abf047d3\") " pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:44.906913 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.906875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdds\" (UniqueName: \"kubernetes.io/projected/fbdc3277-8613-43df-b73c-18c2abf047d3-kube-api-access-9bdds\") pod \"curl-metrics-cc7ec9a4\" (UID: \"fbdc3277-8613-43df-b73c-18c2abf047d3\") " pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:44.914996 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:44.914975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdds\" (UniqueName: \"kubernetes.io/projected/fbdc3277-8613-43df-b73c-18c2abf047d3-kube-api-access-9bdds\") pod \"curl-metrics-cc7ec9a4\" (UID: \"fbdc3277-8613-43df-b73c-18c2abf047d3\") " pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:45.072757 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:45.072738    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:45.340011 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:45.339990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-cc7ec9a4"]
Sep 17 17:57:45.342618 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:57:45.342591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbdc3277_8613_43df_b73c_18c2abf047d3.slice/crio-b521cb682ee8c62e586e0eb73402dcbc506b5c4be8b2b3a6fffa26e8832184d5 WatchSource:0}: Error finding container b521cb682ee8c62e586e0eb73402dcbc506b5c4be8b2b3a6fffa26e8832184d5: Status 404 returned error can't find the container with id b521cb682ee8c62e586e0eb73402dcbc506b5c4be8b2b3a6fffa26e8832184d5
Sep 17 17:57:45.344401 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:45.344381    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:57:45.947861 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:45.947816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-cc7ec9a4" event={"ID":"fbdc3277-8613-43df-b73c-18c2abf047d3","Type":"ContainerStarted","Data":"b521cb682ee8c62e586e0eb73402dcbc506b5c4be8b2b3a6fffa26e8832184d5"}
Sep 17 17:57:46.954349 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:46.954304    2562 generic.go:356] "Generic (PLEG): container finished" podID="fbdc3277-8613-43df-b73c-18c2abf047d3" containerID="b6d216d3ecbd4c48fe030e1ac033c9fa970e5220b40db4ba6736454d64202429" exitCode=0
Sep 17 17:57:46.954721 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:46.954397    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-cc7ec9a4" event={"ID":"fbdc3277-8613-43df-b73c-18c2abf047d3","Type":"ContainerDied","Data":"b6d216d3ecbd4c48fe030e1ac033c9fa970e5220b40db4ba6736454d64202429"}
Sep 17 17:57:48.077019 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.076999    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:48.122528 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.122506    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-cc7ec9a4_fbdc3277-8613-43df-b73c-18c2abf047d3/curl-metrics-cc7ec9a4/0.log"
Sep 17 17:57:48.133812 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.133782    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bdds\" (UniqueName: \"kubernetes.io/projected/fbdc3277-8613-43df-b73c-18c2abf047d3-kube-api-access-9bdds\") pod \"fbdc3277-8613-43df-b73c-18c2abf047d3\" (UID: \"fbdc3277-8613-43df-b73c-18c2abf047d3\") "
Sep 17 17:57:48.135674 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.135654    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdc3277-8613-43df-b73c-18c2abf047d3-kube-api-access-9bdds" (OuterVolumeSpecName: "kube-api-access-9bdds") pod "fbdc3277-8613-43df-b73c-18c2abf047d3" (UID: "fbdc3277-8613-43df-b73c-18c2abf047d3"). InnerVolumeSpecName "kube-api-access-9bdds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:48.147803 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.147775    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-cc7ec9a4"]
Sep 17 17:57:48.151807 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.151788    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-cc7ec9a4"]
Sep 17 17:57:48.235425 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.235362    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bdds\" (UniqueName: \"kubernetes.io/projected/fbdc3277-8613-43df-b73c-18c2abf047d3-kube-api-access-9bdds\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:48.964115 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.964089    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:48.964115 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:48.964113    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b521cb682ee8c62e586e0eb73402dcbc506b5c4be8b2b3a6fffa26e8832184d5"
Sep 17 17:57:48.973625 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:57:48.973602    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-cc7ec9a4\" is forbidden: User \"system:node:ip-10-0-133-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-133-71.ec2.internal' and this object" podUID="fbdc3277-8613-43df-b73c-18c2abf047d3" pod="kserve/curl-metrics-cc7ec9a4"
Sep 17 17:57:49.727069 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:49.727031    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbdc3277-8613-43df-b73c-18c2abf047d3" path="/var/lib/kubelet/pods/fbdc3277-8613-43df-b73c-18c2abf047d3/volumes"
Sep 17 17:57:53.528762 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.528732    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-cac80796"]
Sep 17 17:57:53.529107 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.529046    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdc3277-8613-43df-b73c-18c2abf047d3" containerName="curl-metrics-cc7ec9a4"
Sep 17 17:57:53.529107 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.529057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdc3277-8613-43df-b73c-18c2abf047d3" containerName="curl-metrics-cc7ec9a4"
Sep 17 17:57:53.529107 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.529108    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdc3277-8613-43df-b73c-18c2abf047d3" containerName="curl-metrics-cc7ec9a4"
Sep 17 17:57:53.532057 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.532043    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:53.540096 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.540072    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-cac80796"]
Sep 17 17:57:53.575283 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.575262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbsr\" (UniqueName: \"kubernetes.io/projected/4d1887dc-e769-4234-ab0b-eb5cfd659ed5-kube-api-access-mhbsr\") pod \"curl-metrics-cac80796\" (UID: \"4d1887dc-e769-4234-ab0b-eb5cfd659ed5\") " pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:53.676599 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.676577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbsr\" (UniqueName: \"kubernetes.io/projected/4d1887dc-e769-4234-ab0b-eb5cfd659ed5-kube-api-access-mhbsr\") pod \"curl-metrics-cac80796\" (UID: \"4d1887dc-e769-4234-ab0b-eb5cfd659ed5\") " pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:53.690174 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.690147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbsr\" (UniqueName: \"kubernetes.io/projected/4d1887dc-e769-4234-ab0b-eb5cfd659ed5-kube-api-access-mhbsr\") pod \"curl-metrics-cac80796\" (UID: \"4d1887dc-e769-4234-ab0b-eb5cfd659ed5\") " pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:53.850805 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:53.850785    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:54.126788 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:54.126766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-cac80796"]
Sep 17 17:57:54.129214 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:57:54.129193    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d1887dc_e769_4234_ab0b_eb5cfd659ed5.slice/crio-743b82279d789bb2b33691b257cbd5b95f6296338ead115d8d3a8f5519643560 WatchSource:0}: Error finding container 743b82279d789bb2b33691b257cbd5b95f6296338ead115d8d3a8f5519643560: Status 404 returned error can't find the container with id 743b82279d789bb2b33691b257cbd5b95f6296338ead115d8d3a8f5519643560
Sep 17 17:57:54.992221 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:54.992152    2562 generic.go:356] "Generic (PLEG): container finished" podID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" containerID="95bbcf4e3b733a694dab2ba947869520f6c40b37b6012f3d9e61bdebf7d4e348" exitCode=0
Sep 17 17:57:54.992564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:54.992238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-cac80796" event={"ID":"4d1887dc-e769-4234-ab0b-eb5cfd659ed5","Type":"ContainerDied","Data":"95bbcf4e3b733a694dab2ba947869520f6c40b37b6012f3d9e61bdebf7d4e348"}
Sep 17 17:57:54.992564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:54.992274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-cac80796" event={"ID":"4d1887dc-e769-4234-ab0b-eb5cfd659ed5","Type":"ContainerStarted","Data":"743b82279d789bb2b33691b257cbd5b95f6296338ead115d8d3a8f5519643560"}
Sep 17 17:57:56.114993 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.114969    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:56.137963 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.137941    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-cac80796_4d1887dc-e769-4234-ab0b-eb5cfd659ed5/curl-metrics-cac80796/0.log"
Sep 17 17:57:56.162272 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.162248    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-cac80796"]
Sep 17 17:57:56.165854 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.165830    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-cac80796"]
Sep 17 17:57:56.195649 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.195628    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhbsr\" (UniqueName: \"kubernetes.io/projected/4d1887dc-e769-4234-ab0b-eb5cfd659ed5-kube-api-access-mhbsr\") pod \"4d1887dc-e769-4234-ab0b-eb5cfd659ed5\" (UID: \"4d1887dc-e769-4234-ab0b-eb5cfd659ed5\") "
Sep 17 17:57:56.197585 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.197568    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1887dc-e769-4234-ab0b-eb5cfd659ed5-kube-api-access-mhbsr" (OuterVolumeSpecName: "kube-api-access-mhbsr") pod "4d1887dc-e769-4234-ab0b-eb5cfd659ed5" (UID: "4d1887dc-e769-4234-ab0b-eb5cfd659ed5"). InnerVolumeSpecName "kube-api-access-mhbsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:56.296673 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:56.296623    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbsr\" (UniqueName: \"kubernetes.io/projected/4d1887dc-e769-4234-ab0b-eb5cfd659ed5-kube-api-access-mhbsr\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:57:57.007742 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:57.007708    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:57.007909 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:57.007756    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="743b82279d789bb2b33691b257cbd5b95f6296338ead115d8d3a8f5519643560"
Sep 17 17:57:57.017208 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:57:57.017182    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-cac80796\" is forbidden: User \"system:node:ip-10-0-133-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-133-71.ec2.internal' and this object" podUID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:57.727107 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:57:57.727067    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-cac80796\" is forbidden: User \"system:node:ip-10-0-133-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-133-71.ec2.internal' and this object" podUID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" pod="kserve/curl-metrics-cac80796"
Sep 17 17:57:57.728464 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:57:57.728442    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" path="/var/lib/kubelet/pods/4d1887dc-e769-4234-ab0b-eb5cfd659ed5/volumes"
Sep 17 17:58:01.599440 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.599363    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"]
Sep 17 17:58:01.599772 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.599737    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" containerName="curl-metrics-cac80796"
Sep 17 17:58:01.599772 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.599750    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" containerName="curl-metrics-cac80796"
Sep 17 17:58:01.599852 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.599828    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1887dc-e769-4234-ab0b-eb5cfd659ed5" containerName="curl-metrics-cac80796"
Sep 17 17:58:01.605698 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.605671    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.609592 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.609572    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:58:01.609724 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.609663    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 17:58:01.609906 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.609891    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:58:01.610002 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.609984    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:58:01.610728 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.610710    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:58:01.610728 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.610721    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:58:01.616032 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.614665    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 17:58:01.617399 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.617378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"]
Sep 17 17:58:01.738518 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.738489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kserve-provision-location\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.738755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.738529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f9b42f-eef1-471c-b739-86c7a0638ca1-proxy-tls\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.738755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.738558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsb4h\" (UniqueName: \"kubernetes.io/projected/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-api-access-zsb4h\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.738755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.738628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-cabundle-cert\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.738755 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.738654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.839954 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.839922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb4h\" (UniqueName: \"kubernetes.io/projected/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-api-access-zsb4h\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840114 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.839979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-cabundle-cert\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840114 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.840104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840364 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.840240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kserve-provision-location\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840364 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.840297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f9b42f-eef1-471c-b739-86c7a0638ca1-proxy-tls\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840601 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.840582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kserve-provision-location\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840689 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.840672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-cabundle-cert\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.840802 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.840779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.842910 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.842887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f9b42f-eef1-471c-b739-86c7a0638ca1-proxy-tls\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.848847 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.848826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb4h\" (UniqueName: \"kubernetes.io/projected/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-api-access-zsb4h\") pod \"isvc-sklearn-predictor-9d87cbc64-6xs48\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:01.917346 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:01.917265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:02.190511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:02.190428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"]
Sep 17 17:58:02.192657 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:58:02.192633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f9b42f_eef1_471c_b739_86c7a0638ca1.slice/crio-6b9e9b557cc47bb10cb8205582458938255aba65a7b0c851cac36f44f5d2a14b WatchSource:0}: Error finding container 6b9e9b557cc47bb10cb8205582458938255aba65a7b0c851cac36f44f5d2a14b: Status 404 returned error can't find the container with id 6b9e9b557cc47bb10cb8205582458938255aba65a7b0c851cac36f44f5d2a14b
Sep 17 17:58:03.036573 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:03.036540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerStarted","Data":"812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1"}
Sep 17 17:58:03.036910 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:03.036597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerStarted","Data":"6b9e9b557cc47bb10cb8205582458938255aba65a7b0c851cac36f44f5d2a14b"}
Sep 17 17:58:07.055862 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:07.055834    2562 generic.go:356] "Generic (PLEG): container finished" podID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerID="812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1" exitCode=0
Sep 17 17:58:07.056261 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:07.055904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerDied","Data":"812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1"}
Sep 17 17:58:08.062029 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:08.061996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerStarted","Data":"c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353"}
Sep 17 17:58:08.062444 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:08.062058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerStarted","Data":"dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53"}
Sep 17 17:58:08.062444 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:08.062241    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:08.107699 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:08.105500    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podStartSLOduration=3.30839647 podStartE2EDuration="7.105478832s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797082362s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:08.103396085 +0000 UTC m=+1432.551226559" watchObservedRunningTime="2026-09-17 17:58:08.105478832 +0000 UTC m=+1432.553309287"
Sep 17 17:58:09.066571 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:09.066541    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:09.067802 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:09.067772    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:58:10.070789 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:10.070749    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:58:15.075417 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:15.075392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:58:15.075849 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:15.075824    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:58:25.075965 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:25.075924    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:58:35.076560 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:35.076519    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:58:45.075844 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:45.075799    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:58:55.075861 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:58:55.075822    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:59:05.076620 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:05.076582    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:59:15.076348 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:15.076296    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:59:17.755393 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:17.755305    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:59:17.756349 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:17.756313    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 17:59:25.077114 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:25.077088    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:59:31.748295 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:31.748217    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"]
Sep 17 17:59:31.748672 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:31.748462    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" containerID="cri-o://dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53" gracePeriod=30
Sep 17 17:59:31.748672 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:31.748532    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kube-rbac-proxy" containerID="cri-o://c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353" gracePeriod=30
Sep 17 17:59:32.449405 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:32.449370    2562 generic.go:356] "Generic (PLEG): container finished" podID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerID="c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353" exitCode=2
Sep 17 17:59:32.449602 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:32.449423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerDied","Data":"c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353"}
Sep 17 17:59:35.071465 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.071428    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 17:59:35.075742 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.075721    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 17:59:35.882254 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.882229    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:59:35.907439 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsb4h\" (UniqueName: \"kubernetes.io/projected/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-api-access-zsb4h\") pod \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") "
Sep 17 17:59:35.907564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907463    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-rbac-proxy-sar-config\") pod \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") "
Sep 17 17:59:35.907564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907483    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-cabundle-cert\") pod \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") "
Sep 17 17:59:35.907564 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f9b42f-eef1-471c-b739-86c7a0638ca1-proxy-tls\") pod \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") "
Sep 17 17:59:35.907735 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907599    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kserve-provision-location\") pod \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\" (UID: \"a5f9b42f-eef1-471c-b739-86c7a0638ca1\") "
Sep 17 17:59:35.907902 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907874    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5f9b42f-eef1-471c-b739-86c7a0638ca1" (UID: "a5f9b42f-eef1-471c-b739-86c7a0638ca1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:35.907976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907909    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5f9b42f-eef1-471c-b739-86c7a0638ca1" (UID: "a5f9b42f-eef1-471c-b739-86c7a0638ca1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:35.907976 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.907939    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5f9b42f-eef1-471c-b739-86c7a0638ca1" (UID: "a5f9b42f-eef1-471c-b739-86c7a0638ca1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:35.909759 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.909739    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f9b42f-eef1-471c-b739-86c7a0638ca1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5f9b42f-eef1-471c-b739-86c7a0638ca1" (UID: "a5f9b42f-eef1-471c-b739-86c7a0638ca1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:35.909977 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:35.909950    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-api-access-zsb4h" (OuterVolumeSpecName: "kube-api-access-zsb4h") pod "a5f9b42f-eef1-471c-b739-86c7a0638ca1" (UID: "a5f9b42f-eef1-471c-b739-86c7a0638ca1"). InnerVolumeSpecName "kube-api-access-zsb4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:36.008566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.008472    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsb4h\" (UniqueName: \"kubernetes.io/projected/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-api-access-zsb4h\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:59:36.008566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.008512    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:59:36.008566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.008525    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f9b42f-eef1-471c-b739-86c7a0638ca1-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:59:36.008566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.008536    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f9b42f-eef1-471c-b739-86c7a0638ca1-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:59:36.008566 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.008549    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f9b42f-eef1-471c-b739-86c7a0638ca1-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 17:59:36.468441 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.468394    2562 generic.go:356] "Generic (PLEG): container finished" podID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerID="dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53" exitCode=0
Sep 17 17:59:36.468845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.468468    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"
Sep 17 17:59:36.468845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.468477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerDied","Data":"dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53"}
Sep 17 17:59:36.468845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.468575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48" event={"ID":"a5f9b42f-eef1-471c-b739-86c7a0638ca1","Type":"ContainerDied","Data":"6b9e9b557cc47bb10cb8205582458938255aba65a7b0c851cac36f44f5d2a14b"}
Sep 17 17:59:36.468845 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.468612    2562 scope.go:117] "RemoveContainer" containerID="c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353"
Sep 17 17:59:36.477675 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.477656    2562 scope.go:117] "RemoveContainer" containerID="dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53"
Sep 17 17:59:36.485211 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.485197    2562 scope.go:117] "RemoveContainer" containerID="812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1"
Sep 17 17:59:36.492169 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.492155    2562 scope.go:117] "RemoveContainer" containerID="c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353"
Sep 17 17:59:36.492432 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:59:36.492414    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353\": container with ID starting with c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353 not found: ID does not exist" containerID="c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353"
Sep 17 17:59:36.492506 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.492442    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353"} err="failed to get container status \"c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353\": rpc error: code = NotFound desc = could not find container \"c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353\": container with ID starting with c1d8636fbb2df57a7d1fa34676e2e91b070f5217feca920d3c8e79900671e353 not found: ID does not exist"
Sep 17 17:59:36.492506 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.492459    2562 scope.go:117] "RemoveContainer" containerID="dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53"
Sep 17 17:59:36.492754 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:59:36.492729    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53\": container with ID starting with dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53 not found: ID does not exist" containerID="dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53"
Sep 17 17:59:36.492981 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.492760    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53"} err="failed to get container status \"dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53\": rpc error: code = NotFound desc = could not find container \"dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53\": container with ID starting with dca7735425927561ddd267be0a2a9666868d81c8cbc146005de70795501dff53 not found: ID does not exist"
Sep 17 17:59:36.492981 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.492802    2562 scope.go:117] "RemoveContainer" containerID="812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1"
Sep 17 17:59:36.493199 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:59:36.493180    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1\": container with ID starting with 812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1 not found: ID does not exist" containerID="812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1"
Sep 17 17:59:36.493317 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.493292    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1"} err="failed to get container status \"812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1\": rpc error: code = NotFound desc = could not find container \"812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1\": container with ID starting with 812883b2b127ef712b118ecd4315cb5b6b0effabacfb8662711fc22accb38bb1 not found: ID does not exist"
Sep 17 17:59:36.494870 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.494852    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"]
Sep 17 17:59:36.498460 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.498442    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-9d87cbc64-6xs48"]
Sep 17 17:59:36.874250 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874223    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"]
Sep 17 17:59:36.874633 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874616    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kube-rbac-proxy"
Sep 17 17:59:36.874717 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874636    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kube-rbac-proxy"
Sep 17 17:59:36.874717 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874654    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container"
Sep 17 17:59:36.874717 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874662    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container"
Sep 17 17:59:36.874717 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874696    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="storage-initializer"
Sep 17 17:59:36.874717 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874705    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="storage-initializer"
Sep 17 17:59:36.874966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874783    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kserve-container"
Sep 17 17:59:36.874966 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.874801    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" containerName="kube-rbac-proxy"
Sep 17 17:59:36.879465 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.879445    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:36.882632 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.882605    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:59:36.882738 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.882673    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:59:36.882937 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.882915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:59:36.883077 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.882965    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:59:36.883201 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.883184    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 17:59:36.883827 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.883810    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:59:36.884022 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.884006    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 17:59:36.894023 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.894001    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"]
Sep 17 17:59:36.916170 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.916150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:36.916284 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.916204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:36.916284 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.916226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:36.916284 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.916243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:36.916284 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:36.916272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljqw\" (UniqueName: \"kubernetes.io/projected/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-api-access-xljqw\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017145 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017246 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017297 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017373 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:59:37.017304    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 17:59:37.017373 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017463 ip-10-0-133-71 kubenswrapper[2562]: E0917 17:59:37.017402    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls podName:5f2b92e3-9dbf-4aac-9a34-df5914a06e14 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:37.517381725 +0000 UTC m=+1521.965212173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" (UID: "5f2b92e3-9dbf-4aac-9a34-df5914a06e14") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 17:59:37.017463 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljqw\" (UniqueName: \"kubernetes.io/projected/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-api-access-xljqw\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017650 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017917 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.017959 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.017898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.026080 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.026056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljqw\" (UniqueName: \"kubernetes.io/projected/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-api-access-xljqw\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.521471 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.521439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.523797 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.523780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:37.726749 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.726720    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f9b42f-eef1-471c-b739-86c7a0638ca1" path="/var/lib/kubelet/pods/a5f9b42f-eef1-471c-b739-86c7a0638ca1/volumes"
Sep 17 17:59:37.789193 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:37.789116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 17:59:38.067511 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:38.067481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"]
Sep 17 17:59:38.070233 ip-10-0-133-71 kubenswrapper[2562]: W0917 17:59:38.070195    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2b92e3_9dbf_4aac_9a34_df5914a06e14.slice/crio-f0fe904e5d931be28dc15c6669b6b5cce6cf99a2515f1dfaeb6adb0cc25e6ab8 WatchSource:0}: Error finding container f0fe904e5d931be28dc15c6669b6b5cce6cf99a2515f1dfaeb6adb0cc25e6ab8: Status 404 returned error can't find the container with id f0fe904e5d931be28dc15c6669b6b5cce6cf99a2515f1dfaeb6adb0cc25e6ab8
Sep 17 17:59:38.481270 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:38.481193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerStarted","Data":"47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781"}
Sep 17 17:59:38.481270 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:38.481257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerStarted","Data":"f0fe904e5d931be28dc15c6669b6b5cce6cf99a2515f1dfaeb6adb0cc25e6ab8"}
Sep 17 17:59:42.499848 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:42.499808    2562 generic.go:356] "Generic (PLEG): container finished" podID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerID="47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781" exitCode=0
Sep 17 17:59:42.500316 ip-10-0-133-71 kubenswrapper[2562]: I0917 17:59:42.499873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerDied","Data":"47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781"}
Sep 17 18:01:51.206539 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:01:51.206506    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerStarted","Data":"f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728"}
Sep 17 18:01:51.206990 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:01:51.206574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerStarted","Data":"be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051"}
Sep 17 18:01:51.206990 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:01:51.206611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 18:01:51.206990 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:01:51.206633    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 18:01:51.244439 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:01:51.244378    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podStartSLOduration=3.229975562 podStartE2EDuration="2m15.244358235s" totalImagesPullingTime="2m8.276952979s" totalInitContainerRuntime="3.737429694s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:51.242593655 +0000 UTC m=+1655.690424150" watchObservedRunningTime="2026-09-17 18:01:51.244358235 +0000 UTC m=+1655.692188691"
Sep 17 18:01:57.216211 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:01:57.216180    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 18:02:27.320601 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:27.320554    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:02:37.219127 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:37.219053    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 18:02:47.125708 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:47.125672    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"]
Sep 17 18:02:47.126106 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:47.125938    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kserve-container" containerID="cri-o://be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051" gracePeriod=30
Sep 17 18:02:47.126106 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:47.125990    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kube-rbac-proxy" containerID="cri-o://f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728" gracePeriod=30
Sep 17 18:02:47.211982 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:47.211947    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 18:02:47.472751 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:47.472660    2562 generic.go:356] "Generic (PLEG): container finished" podID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerID="f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728" exitCode=2
Sep 17 18:02:47.472751 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:47.472720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerDied","Data":"f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728"}
Sep 17 18:02:48.257508 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:48.257468    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:02:52.211591 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.211549    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 18:02:52.366135 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.366102    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"]
Sep 17 18:02:52.368572 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.368557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.372964 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.372945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 18:02:52.373137 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.373123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:02:52.406388 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.406359    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"]
Sep 17 18:02:52.421948 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.421922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56w7\" (UniqueName: \"kubernetes.io/projected/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-api-access-n56w7\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.422101 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.422016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.422101 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.422046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.422224 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.422116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a674ebac-cc54-47dd-ac0b-4d4599fb6991-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.422224 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.422161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.523117 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.523034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56w7\" (UniqueName: \"kubernetes.io/projected/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-api-access-n56w7\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.523117 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.523092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.523366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.523122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.523366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.523166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a674ebac-cc54-47dd-ac0b-4d4599fb6991-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.523366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.523208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.523591 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.523569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.524054 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.524031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.524099 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.524031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.525608 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.525581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a674ebac-cc54-47dd-ac0b-4d4599fb6991-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.540574 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.540550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56w7\" (UniqueName: \"kubernetes.io/projected/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-api-access-n56w7\") pod \"isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.678557 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.678511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:02:52.963115 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:52.963088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"]
Sep 17 18:02:52.965000 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:02:52.964977    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda674ebac_cc54_47dd_ac0b_4d4599fb6991.slice/crio-f47a9b68dbad1e46d461d186ce44f71e302203e714c1ce5a2dd704047a3e35f9 WatchSource:0}: Error finding container f47a9b68dbad1e46d461d186ce44f71e302203e714c1ce5a2dd704047a3e35f9: Status 404 returned error can't find the container with id f47a9b68dbad1e46d461d186ce44f71e302203e714c1ce5a2dd704047a3e35f9
Sep 17 18:02:53.502432 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:53.502391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerStarted","Data":"81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e"}
Sep 17 18:02:53.502811 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:53.502466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerStarted","Data":"f47a9b68dbad1e46d461d186ce44f71e302203e714c1ce5a2dd704047a3e35f9"}
Sep 17 18:02:54.269030 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.269009    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 18:02:54.339661 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.339627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-rbac-proxy-sar-config\") pod \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") "
Sep 17 18:02:54.339816 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.339713    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-cabundle-cert\") pod \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") "
Sep 17 18:02:54.339816 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.339806    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kserve-provision-location\") pod \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") "
Sep 17 18:02:54.339937 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.339840    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls\") pod \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") "
Sep 17 18:02:54.339937 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.339872    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xljqw\" (UniqueName: \"kubernetes.io/projected/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-api-access-xljqw\") pod \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\" (UID: \"5f2b92e3-9dbf-4aac-9a34-df5914a06e14\") "
Sep 17 18:02:54.340089 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.340066    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f2b92e3-9dbf-4aac-9a34-df5914a06e14" (UID: "5f2b92e3-9dbf-4aac-9a34-df5914a06e14"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:54.340157 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.340076    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f2b92e3-9dbf-4aac-9a34-df5914a06e14" (UID: "5f2b92e3-9dbf-4aac-9a34-df5914a06e14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:54.340157 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.340108    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f2b92e3-9dbf-4aac-9a34-df5914a06e14" (UID: "5f2b92e3-9dbf-4aac-9a34-df5914a06e14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:54.340255 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.340185    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.340255 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.340201    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.340255 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.340214    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.342043 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.342020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f2b92e3-9dbf-4aac-9a34-df5914a06e14" (UID: "5f2b92e3-9dbf-4aac-9a34-df5914a06e14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:54.342186 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.342163    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-api-access-xljqw" (OuterVolumeSpecName: "kube-api-access-xljqw") pod "5f2b92e3-9dbf-4aac-9a34-df5914a06e14" (UID: "5f2b92e3-9dbf-4aac-9a34-df5914a06e14"). InnerVolumeSpecName "kube-api-access-xljqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:54.441509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.441422    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljqw\" (UniqueName: \"kubernetes.io/projected/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-kube-api-access-xljqw\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.441509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.441460    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2b92e3-9dbf-4aac-9a34-df5914a06e14-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.509313 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.509281    2562 generic.go:356] "Generic (PLEG): container finished" podID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerID="be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051" exitCode=0
Sep 17 18:02:54.509741 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.509346    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"
Sep 17 18:02:54.509741 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.509362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerDied","Data":"be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051"}
Sep 17 18:02:54.509741 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.509414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48" event={"ID":"5f2b92e3-9dbf-4aac-9a34-df5914a06e14","Type":"ContainerDied","Data":"f0fe904e5d931be28dc15c6669b6b5cce6cf99a2515f1dfaeb6adb0cc25e6ab8"}
Sep 17 18:02:54.509741 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.509440    2562 scope.go:117] "RemoveContainer" containerID="f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728"
Sep 17 18:02:54.518544 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.518530    2562 scope.go:117] "RemoveContainer" containerID="be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051"
Sep 17 18:02:54.526140 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.526123    2562 scope.go:117] "RemoveContainer" containerID="47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781"
Sep 17 18:02:54.534702 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.534688    2562 scope.go:117] "RemoveContainer" containerID="f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728"
Sep 17 18:02:54.534992 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:02:54.534975    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728\": container with ID starting with f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728 not found: ID does not exist" containerID="f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728"
Sep 17 18:02:54.535075 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.534998    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728"} err="failed to get container status \"f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728\": rpc error: code = NotFound desc = could not find container \"f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728\": container with ID starting with f341c801d24d9bf84fa0d0c2c72ccc217c407dc850823baa5194574dbd19a728 not found: ID does not exist"
Sep 17 18:02:54.535075 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.535013    2562 scope.go:117] "RemoveContainer" containerID="be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051"
Sep 17 18:02:54.535280 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:02:54.535253    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051\": container with ID starting with be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051 not found: ID does not exist" containerID="be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051"
Sep 17 18:02:54.535317 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.535286    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051"} err="failed to get container status \"be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051\": rpc error: code = NotFound desc = could not find container \"be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051\": container with ID starting with be30027183405b099d9e9418b3719bbeb78cc58d3e8b08271bdafd7763182051 not found: ID does not exist"
Sep 17 18:02:54.535317 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.535302    2562 scope.go:117] "RemoveContainer" containerID="47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781"
Sep 17 18:02:54.535597 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:02:54.535579    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781\": container with ID starting with 47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781 not found: ID does not exist" containerID="47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781"
Sep 17 18:02:54.535667 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.535601    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781"} err="failed to get container status \"47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781\": rpc error: code = NotFound desc = could not find container \"47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781\": container with ID starting with 47e10ce9ee43384aa5904d6daa27acf7bcf18e544d8734a52d56e3c0110c7781 not found: ID does not exist"
Sep 17 18:02:54.538156 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.538134    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"]
Sep 17 18:02:54.542872 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:54.542852    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-k6c48"]
Sep 17 18:02:55.726919 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:55.726888    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" path="/var/lib/kubelet/pods/5f2b92e3-9dbf-4aac-9a34-df5914a06e14/volumes"
Sep 17 18:02:58.307397 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307361    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"]
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307693    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="storage-initializer"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307704    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="storage-initializer"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307714    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kserve-container"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307719    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kserve-container"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307738    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kube-rbac-proxy"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307744    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kube-rbac-proxy"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307810    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kube-rbac-proxy"
Sep 17 18:02:58.307820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.307819    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2b92e3-9dbf-4aac-9a34-df5914a06e14" containerName="kserve-container"
Sep 17 18:02:58.310843 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.310822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.313830 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.313812    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:02:58.313934 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.313817    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:02:58.331675 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.331644    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"]
Sep 17 18:02:58.373191 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.373167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.373366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.373221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.373366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.373253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebe2dd6-2e33-404b-aefd-db962812555a-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.373366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.373270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk592\" (UniqueName: \"kubernetes.io/projected/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-api-access-qk592\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.373366 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.373357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2dd6-2e33-404b-aefd-db962812555a-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.473811 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.473772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk592\" (UniqueName: \"kubernetes.io/projected/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-api-access-qk592\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.473836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2dd6-2e33-404b-aefd-db962812555a-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.473877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.473944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.473994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebe2dd6-2e33-404b-aefd-db962812555a-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474490 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.474465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2dd6-2e33-404b-aefd-db962812555a-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474751 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.474727    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.474803 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.474788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.476527 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.476508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebe2dd6-2e33-404b-aefd-db962812555a-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.485321 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.485301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk592\" (UniqueName: \"kubernetes.io/projected/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-api-access-qk592\") pod \"isvc-canary-create-predictor-6474b875b9-v4kd2\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.621858 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.621820    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:02:58.917275 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:58.917188    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"]
Sep 17 18:02:58.921083 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:02:58.921056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebe2dd6_2e33_404b_aefd_db962812555a.slice/crio-c8bb32f9a4ece2c21df5f1129e12116611189c0133ac61102705f1471bd4abee WatchSource:0}: Error finding container c8bb32f9a4ece2c21df5f1129e12116611189c0133ac61102705f1471bd4abee: Status 404 returned error can't find the container with id c8bb32f9a4ece2c21df5f1129e12116611189c0133ac61102705f1471bd4abee
Sep 17 18:02:59.538361 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:59.538315    2562 generic.go:356] "Generic (PLEG): container finished" podID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerID="81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e" exitCode=0
Sep 17 18:02:59.538820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:59.538389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerDied","Data":"81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e"}
Sep 17 18:02:59.540377 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:59.540350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerStarted","Data":"8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d"}
Sep 17 18:02:59.540463 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:02:59.540413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerStarted","Data":"c8bb32f9a4ece2c21df5f1129e12116611189c0133ac61102705f1471bd4abee"}
Sep 17 18:03:00.548396 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:00.548362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerStarted","Data":"018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52"}
Sep 17 18:03:00.548811 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:00.548423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerStarted","Data":"d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538"}
Sep 17 18:03:00.548811 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:00.548687    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:03:00.548811 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:00.548724    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:03:00.550318 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:00.550291    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:03:00.593090 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:00.593044    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" podStartSLOduration=2.592027216 podStartE2EDuration="8.593030142s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.001002926s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:00.592680192 +0000 UTC m=+1725.040510647" watchObservedRunningTime="2026-09-17 18:03:00.593030142 +0000 UTC m=+1725.040860598"
Sep 17 18:03:01.553755 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:01.553715    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:03:03.565489 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:03.565458    2562 generic.go:356] "Generic (PLEG): container finished" podID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerID="8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d" exitCode=0
Sep 17 18:03:03.565920 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:03.565515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerDied","Data":"8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d"}
Sep 17 18:03:04.572667 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:04.572629    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerStarted","Data":"a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8"}
Sep 17 18:03:04.573255 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:04.572690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerStarted","Data":"bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819"}
Sep 17 18:03:04.573255 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:04.572877    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:03:04.599804 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:04.599748    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podStartSLOduration=2.571835306 podStartE2EDuration="6.599730385s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.027895079s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:04.597952114 +0000 UTC m=+1729.045782590" watchObservedRunningTime="2026-09-17 18:03:04.599730385 +0000 UTC m=+1729.047560840"
Sep 17 18:03:05.578709 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:05.578676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:03:05.580209 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:05.580180    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:03:06.558302 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:06.558270    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:03:06.558950 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:06.558922    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:03:06.590439 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:06.590381    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:03:11.591908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:11.591876    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:03:11.592494 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:11.592467    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:03:16.559848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:16.559817    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:03:21.592733 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:21.592693    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:03:24.219571 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:24.219542    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"]
Sep 17 18:03:24.219991 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:24.219760    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container" containerID="cri-o://d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538" gracePeriod=30
Sep 17 18:03:24.219991 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:24.219819    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kube-rbac-proxy" containerID="cri-o://018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52" gracePeriod=30
Sep 17 18:03:24.682678 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:24.682639    2562 generic.go:356] "Generic (PLEG): container finished" podID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerID="018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52" exitCode=2
Sep 17 18:03:24.682856 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:24.682714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerDied","Data":"018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52"}
Sep 17 18:03:25.161501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.161479    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:03:25.321033 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321002    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n56w7\" (UniqueName: \"kubernetes.io/projected/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-api-access-n56w7\") pod \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") "
Sep 17 18:03:25.321623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321075    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-cabundle-cert\") pod \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") "
Sep 17 18:03:25.321623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321101    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-rbac-proxy-sar-config\") pod \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") "
Sep 17 18:03:25.321623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kserve-provision-location\") pod \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") "
Sep 17 18:03:25.321623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321233    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a674ebac-cc54-47dd-ac0b-4d4599fb6991-proxy-tls\") pod \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\" (UID: \"a674ebac-cc54-47dd-ac0b-4d4599fb6991\") "
Sep 17 18:03:25.321623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321557    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a674ebac-cc54-47dd-ac0b-4d4599fb6991" (UID: "a674ebac-cc54-47dd-ac0b-4d4599fb6991"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:25.321623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.321565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a674ebac-cc54-47dd-ac0b-4d4599fb6991" (UID: "a674ebac-cc54-47dd-ac0b-4d4599fb6991"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:25.323166 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.323145    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-api-access-n56w7" (OuterVolumeSpecName: "kube-api-access-n56w7") pod "a674ebac-cc54-47dd-ac0b-4d4599fb6991" (UID: "a674ebac-cc54-47dd-ac0b-4d4599fb6991"). InnerVolumeSpecName "kube-api-access-n56w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:25.323630 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.323611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a674ebac-cc54-47dd-ac0b-4d4599fb6991-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a674ebac-cc54-47dd-ac0b-4d4599fb6991" (UID: "a674ebac-cc54-47dd-ac0b-4d4599fb6991"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:25.350368 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.350309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a674ebac-cc54-47dd-ac0b-4d4599fb6991" (UID: "a674ebac-cc54-47dd-ac0b-4d4599fb6991"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:25.422866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.422829    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a674ebac-cc54-47dd-ac0b-4d4599fb6991-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:03:25.422866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.422860    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56w7\" (UniqueName: \"kubernetes.io/projected/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-api-access-n56w7\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:03:25.422866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.422872    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:03:25.423088 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.422887    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:03:25.423088 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.422897    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a674ebac-cc54-47dd-ac0b-4d4599fb6991-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:03:25.689315 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.689218    2562 generic.go:356] "Generic (PLEG): container finished" podID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerID="d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538" exitCode=0
Sep 17 18:03:25.689315 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.689289    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"
Sep 17 18:03:25.689315 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.689301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerDied","Data":"d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538"}
Sep 17 18:03:25.689613 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.689381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps" event={"ID":"a674ebac-cc54-47dd-ac0b-4d4599fb6991","Type":"ContainerDied","Data":"f47a9b68dbad1e46d461d186ce44f71e302203e714c1ce5a2dd704047a3e35f9"}
Sep 17 18:03:25.689613 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.689411    2562 scope.go:117] "RemoveContainer" containerID="018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52"
Sep 17 18:03:25.698942 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.698923    2562 scope.go:117] "RemoveContainer" containerID="d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538"
Sep 17 18:03:25.706945 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.706927    2562 scope.go:117] "RemoveContainer" containerID="81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e"
Sep 17 18:03:25.714110 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.714080    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"]
Sep 17 18:03:25.714933 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.714908    2562 scope.go:117] "RemoveContainer" containerID="018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52"
Sep 17 18:03:25.715352 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:03:25.715310    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52\": container with ID starting with 018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52 not found: ID does not exist" containerID="018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52"
Sep 17 18:03:25.715487 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.715365    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52"} err="failed to get container status \"018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52\": rpc error: code = NotFound desc = could not find container \"018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52\": container with ID starting with 018fde1144416500587e98fce4424bde3c65aae3d2b5abaeb4bb71949027ac52 not found: ID does not exist"
Sep 17 18:03:25.715487 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.715390    2562 scope.go:117] "RemoveContainer" containerID="d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538"
Sep 17 18:03:25.715890 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:03:25.715769    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538\": container with ID starting with d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538 not found: ID does not exist" containerID="d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538"
Sep 17 18:03:25.715890 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.715800    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538"} err="failed to get container status \"d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538\": rpc error: code = NotFound desc = could not find container \"d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538\": container with ID starting with d8a2c0b629a899dd16ed6dc4ba04858dce1232cb6153ca4a0037875d362f4538 not found: ID does not exist"
Sep 17 18:03:25.715890 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.715859    2562 scope.go:117] "RemoveContainer" containerID="81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e"
Sep 17 18:03:25.716203 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:03:25.716162    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e\": container with ID starting with 81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e not found: ID does not exist" containerID="81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e"
Sep 17 18:03:25.716264 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.716209    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e"} err="failed to get container status \"81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e\": rpc error: code = NotFound desc = could not find container \"81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e\": container with ID starting with 81ed222a2d1eb74f91bd380704aa5099022b4c54485665b9e329edf675ff293e not found: ID does not exist"
Sep 17 18:03:25.717185 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.717164    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6fc7d5d5bd-9tmps"]
Sep 17 18:03:25.729553 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:25.727060    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" path="/var/lib/kubelet/pods/a674ebac-cc54-47dd-ac0b-4d4599fb6991/volumes"
Sep 17 18:03:29.626960 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.626924    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"]
Sep 17 18:03:29.627374 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627300    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container"
Sep 17 18:03:29.627374 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627311    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container"
Sep 17 18:03:29.627374 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627322    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kube-rbac-proxy"
Sep 17 18:03:29.627374 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627340    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kube-rbac-proxy"
Sep 17 18:03:29.627374 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627349    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="storage-initializer"
Sep 17 18:03:29.627374 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627355    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="storage-initializer"
Sep 17 18:03:29.627575 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627405    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kserve-container"
Sep 17 18:03:29.627575 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.627412    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a674ebac-cc54-47dd-ac0b-4d4599fb6991" containerName="kube-rbac-proxy"
Sep 17 18:03:29.630302 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.630286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.636052 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.636012    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 18:03:29.636218 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.636122    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:03:29.667926 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.667887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"]
Sep 17 18:03:29.765133 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.765097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.765133 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.765134    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36806458-a53b-4cde-9b99-daa98004b58d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.765389 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.765160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36806458-a53b-4cde-9b99-daa98004b58d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.765389 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.765204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.765389 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.765318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wfk5\" (UniqueName: \"kubernetes.io/projected/36806458-a53b-4cde-9b99-daa98004b58d-kube-api-access-7wfk5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.866437 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.866394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wfk5\" (UniqueName: \"kubernetes.io/projected/36806458-a53b-4cde-9b99-daa98004b58d-kube-api-access-7wfk5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.866650 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.866467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.866650 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.866486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36806458-a53b-4cde-9b99-daa98004b58d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.866650 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.866516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36806458-a53b-4cde-9b99-daa98004b58d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.866650 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.866532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.866948 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.866926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36806458-a53b-4cde-9b99-daa98004b58d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.867181 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.867162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.867242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.867179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.869085 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.869060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36806458-a53b-4cde-9b99-daa98004b58d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.881257 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.881197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wfk5\" (UniqueName: \"kubernetes.io/projected/36806458-a53b-4cde-9b99-daa98004b58d-kube-api-access-7wfk5\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:29.941156 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:29.941116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:30.232202 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:30.232054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"]
Sep 17 18:03:30.234863 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:03:30.234835    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36806458_a53b_4cde_9b99_daa98004b58d.slice/crio-b2272d4b7456279b6ca9d7249b76e52554faa1cab1f3e424c8902b2f97dcd4d9 WatchSource:0}: Error finding container b2272d4b7456279b6ca9d7249b76e52554faa1cab1f3e424c8902b2f97dcd4d9: Status 404 returned error can't find the container with id b2272d4b7456279b6ca9d7249b76e52554faa1cab1f3e424c8902b2f97dcd4d9
Sep 17 18:03:30.715602 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:30.715567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerStarted","Data":"a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650"}
Sep 17 18:03:30.715976 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:30.715629    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerStarted","Data":"b2272d4b7456279b6ca9d7249b76e52554faa1cab1f3e424c8902b2f97dcd4d9"}
Sep 17 18:03:31.592549 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:31.592513    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:03:34.736454 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:34.736414    2562 generic.go:356] "Generic (PLEG): container finished" podID="36806458-a53b-4cde-9b99-daa98004b58d" containerID="a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650" exitCode=0
Sep 17 18:03:34.736874 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:34.736460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerDied","Data":"a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650"}
Sep 17 18:03:35.744390 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:35.744356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerStarted","Data":"960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55"}
Sep 17 18:03:35.744840 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:35.744419    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerStarted","Data":"bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573"}
Sep 17 18:03:35.744840 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:35.744601    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:35.771206 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:35.771153    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" podStartSLOduration=2.9813740859999998 podStartE2EDuration="6.771136342s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789762256s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:35.770760392 +0000 UTC m=+1760.218590877" watchObservedRunningTime="2026-09-17 18:03:35.771136342 +0000 UTC m=+1760.218966797"
Sep 17 18:03:36.748912 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:36.748881    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:41.592919 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:41.592877    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:03:42.759432 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:42.759400    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:03:51.592587 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:03:51.592549    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:04:01.592866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:01.592780    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:04:11.592457 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:11.592411    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:04:12.819767 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:12.819723    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:04:17.788663 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:17.788590    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:04:17.789265 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:17.789243    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:04:17.816753 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:17.816728    2562 scope.go:117] "RemoveContainer" containerID="b6d216d3ecbd4c48fe030e1ac033c9fa970e5220b40db4ba6736454d64202429"
Sep 17 18:04:17.826130 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:17.826109    2562 scope.go:117] "RemoveContainer" containerID="95bbcf4e3b733a694dab2ba947869520f6c40b37b6012f3d9e61bdebf7d4e348"
Sep 17 18:04:21.593412 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:21.593383    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:04:22.762747 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:22.762714    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:04:29.511678 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:29.511647    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"]
Sep 17 18:04:29.512076 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:29.511914    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kserve-container" containerID="cri-o://bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573" gracePeriod=30
Sep 17 18:04:29.512076 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:29.511943    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kube-rbac-proxy" containerID="cri-o://960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55" gracePeriod=30
Sep 17 18:04:30.006372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:30.006320    2562 generic.go:356] "Generic (PLEG): container finished" podID="36806458-a53b-4cde-9b99-daa98004b58d" containerID="960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55" exitCode=2
Sep 17 18:04:30.006372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:30.006358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerDied","Data":"960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55"}
Sep 17 18:04:32.754967 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:32.754926    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 18:04:34.705380 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.705341    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"]
Sep 17 18:04:34.708154 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.708135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.710629 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.710606    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 18:04:34.710916 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.710896    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:04:34.727119 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.727086    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"]
Sep 17 18:04:34.818479 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.818447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d5a47aa-3e4d-47ee-9940-69e118659de0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.818663 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.818530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqc9\" (UniqueName: \"kubernetes.io/projected/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-api-access-gxqc9\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.818663 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.818580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.818663 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.818611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.818663 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.818627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5a47aa-3e4d-47ee-9940-69e118659de0-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.919843 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.919812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqc9\" (UniqueName: \"kubernetes.io/projected/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-api-access-gxqc9\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920039 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.919851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920039 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.919880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920152 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.920062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5a47aa-3e4d-47ee-9940-69e118659de0-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920152 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.920111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d5a47aa-3e4d-47ee-9940-69e118659de0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920447 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.920426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5a47aa-3e4d-47ee-9940-69e118659de0-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920701 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.920678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.920785 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.920678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.922604 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.922584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d5a47aa-3e4d-47ee-9940-69e118659de0-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:34.930930 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:34.930912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqc9\" (UniqueName: \"kubernetes.io/projected/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-api-access-gxqc9\") pod \"isvc-sklearn-v2-predictor-ffc44684-gns9s\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:35.018095 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:35.018022    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:35.295286 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:35.295260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"]
Sep 17 18:04:35.296860 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:04:35.296836    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5a47aa_3e4d_47ee_9940_69e118659de0.slice/crio-a11644ad0b50a32b88710202264071c034a349c7327dc8f2f3195ae6326b20ca WatchSource:0}: Error finding container a11644ad0b50a32b88710202264071c034a349c7327dc8f2f3195ae6326b20ca: Status 404 returned error can't find the container with id a11644ad0b50a32b88710202264071c034a349c7327dc8f2f3195ae6326b20ca
Sep 17 18:04:36.038375 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.038318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerStarted","Data":"20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576"}
Sep 17 18:04:36.038760 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.038403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerStarted","Data":"a11644ad0b50a32b88710202264071c034a349c7327dc8f2f3195ae6326b20ca"}
Sep 17 18:04:36.861282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.861257    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:04:36.936529 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-kube-rbac-proxy-sar-config\") pod \"36806458-a53b-4cde-9b99-daa98004b58d\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") "
Sep 17 18:04:36.936529 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wfk5\" (UniqueName: \"kubernetes.io/projected/36806458-a53b-4cde-9b99-daa98004b58d-kube-api-access-7wfk5\") pod \"36806458-a53b-4cde-9b99-daa98004b58d\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") "
Sep 17 18:04:36.936742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936531    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-cabundle-cert\") pod \"36806458-a53b-4cde-9b99-daa98004b58d\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") "
Sep 17 18:04:36.936742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936555    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36806458-a53b-4cde-9b99-daa98004b58d-kserve-provision-location\") pod \"36806458-a53b-4cde-9b99-daa98004b58d\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") "
Sep 17 18:04:36.936742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936592    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36806458-a53b-4cde-9b99-daa98004b58d-proxy-tls\") pod \"36806458-a53b-4cde-9b99-daa98004b58d\" (UID: \"36806458-a53b-4cde-9b99-daa98004b58d\") "
Sep 17 18:04:36.936906 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936880    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36806458-a53b-4cde-9b99-daa98004b58d" (UID: "36806458-a53b-4cde-9b99-daa98004b58d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:36.936950 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936923    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36806458-a53b-4cde-9b99-daa98004b58d" (UID: "36806458-a53b-4cde-9b99-daa98004b58d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:36.936950 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.936928    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36806458-a53b-4cde-9b99-daa98004b58d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36806458-a53b-4cde-9b99-daa98004b58d" (UID: "36806458-a53b-4cde-9b99-daa98004b58d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:36.938669 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.938647    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36806458-a53b-4cde-9b99-daa98004b58d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36806458-a53b-4cde-9b99-daa98004b58d" (UID: "36806458-a53b-4cde-9b99-daa98004b58d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:36.938788 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:36.938719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36806458-a53b-4cde-9b99-daa98004b58d-kube-api-access-7wfk5" (OuterVolumeSpecName: "kube-api-access-7wfk5") pod "36806458-a53b-4cde-9b99-daa98004b58d" (UID: "36806458-a53b-4cde-9b99-daa98004b58d"). InnerVolumeSpecName "kube-api-access-7wfk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:37.038167 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.038133    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36806458-a53b-4cde-9b99-daa98004b58d-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:37.038167 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.038162    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:37.038167 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.038172    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wfk5\" (UniqueName: \"kubernetes.io/projected/36806458-a53b-4cde-9b99-daa98004b58d-kube-api-access-7wfk5\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:37.038427 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.038181    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36806458-a53b-4cde-9b99-daa98004b58d-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:37.038427 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.038191    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36806458-a53b-4cde-9b99-daa98004b58d-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:37.044786 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.044760    2562 generic.go:356] "Generic (PLEG): container finished" podID="36806458-a53b-4cde-9b99-daa98004b58d" containerID="bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573" exitCode=0
Sep 17 18:04:37.044924 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.044811    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"
Sep 17 18:04:37.044924 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.044837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerDied","Data":"bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573"}
Sep 17 18:04:37.044924 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.044893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7" event={"ID":"36806458-a53b-4cde-9b99-daa98004b58d","Type":"ContainerDied","Data":"b2272d4b7456279b6ca9d7249b76e52554faa1cab1f3e424c8902b2f97dcd4d9"}
Sep 17 18:04:37.044924 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.044922    2562 scope.go:117] "RemoveContainer" containerID="960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55"
Sep 17 18:04:37.054378 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.054361    2562 scope.go:117] "RemoveContainer" containerID="bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573"
Sep 17 18:04:37.061945 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.061929    2562 scope.go:117] "RemoveContainer" containerID="a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650"
Sep 17 18:04:37.069303 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.069287    2562 scope.go:117] "RemoveContainer" containerID="960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55"
Sep 17 18:04:37.069645 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:37.069621    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55\": container with ID starting with 960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55 not found: ID does not exist" containerID="960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55"
Sep 17 18:04:37.069748 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.069656    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55"} err="failed to get container status \"960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55\": rpc error: code = NotFound desc = could not find container \"960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55\": container with ID starting with 960e0274f6583d47a4bb625b2da0b43a25d6e2aef4f62e99bb588fba644c5a55 not found: ID does not exist"
Sep 17 18:04:37.069748 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.069674    2562 scope.go:117] "RemoveContainer" containerID="bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573"
Sep 17 18:04:37.069870 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.069852    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"]
Sep 17 18:04:37.069961 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:37.069946    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573\": container with ID starting with bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573 not found: ID does not exist" containerID="bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573"
Sep 17 18:04:37.070008 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.069965    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573"} err="failed to get container status \"bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573\": rpc error: code = NotFound desc = could not find container \"bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573\": container with ID starting with bc66583be6e60b079fe63bdc3133ccd4a395a645fe9f49127e08962602c69573 not found: ID does not exist"
Sep 17 18:04:37.070008 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.069979    2562 scope.go:117] "RemoveContainer" containerID="a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650"
Sep 17 18:04:37.070193 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:37.070177    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650\": container with ID starting with a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650 not found: ID does not exist" containerID="a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650"
Sep 17 18:04:37.070240 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.070198    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650"} err="failed to get container status \"a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650\": rpc error: code = NotFound desc = could not find container \"a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650\": container with ID starting with a692ed7d1e149afa74c83f1e4b0a75c00d5374f3d942a8f39174dd5c69a5f650 not found: ID does not exist"
Sep 17 18:04:37.075049 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.075029    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7svr7"]
Sep 17 18:04:37.726793 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:37.726763    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36806458-a53b-4cde-9b99-daa98004b58d" path="/var/lib/kubelet/pods/36806458-a53b-4cde-9b99-daa98004b58d/volumes"
Sep 17 18:04:38.324264 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:38.324126    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"]
Sep 17 18:04:38.324739 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:38.324437    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" containerID="cri-o://bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819" gracePeriod=30
Sep 17 18:04:38.324739 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:38.324478    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kube-rbac-proxy" containerID="cri-o://a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8" gracePeriod=30
Sep 17 18:04:39.059906 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:39.059870    2562 generic.go:356] "Generic (PLEG): container finished" podID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerID="a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8" exitCode=2
Sep 17 18:04:39.060107 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:39.059918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerDied","Data":"a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8"}
Sep 17 18:04:40.066184 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:40.066153    2562 generic.go:356] "Generic (PLEG): container finished" podID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerID="20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576" exitCode=0
Sep 17 18:04:40.066706 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:40.066224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerDied","Data":"20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576"}
Sep 17 18:04:41.073606 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.073573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerStarted","Data":"f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542"}
Sep 17 18:04:41.074038 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.073634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerStarted","Data":"13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683"}
Sep 17 18:04:41.074038 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.073919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:41.074156 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.074084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:41.075340 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.075299    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:04:41.104442 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.104401    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podStartSLOduration=3.1203183389999998 podStartE2EDuration="7.104387699s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.98406936s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:41.103542798 +0000 UTC m=+1825.551373272" watchObservedRunningTime="2026-09-17 18:04:41.104387699 +0000 UTC m=+1825.552218153"
Sep 17 18:04:41.587034 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.586998    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 18:04:41.592373 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:41.592347    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:04:42.078462 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:42.078425    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:04:42.979725 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:42.979697    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:04:43.084611 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.084522    2562 generic.go:356] "Generic (PLEG): container finished" podID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerID="bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819" exitCode=0
Sep 17 18:04:43.084611 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.084589    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"
Sep 17 18:04:43.085059 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.084603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerDied","Data":"bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819"}
Sep 17 18:04:43.085059 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.084662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2" event={"ID":"1ebe2dd6-2e33-404b-aefd-db962812555a","Type":"ContainerDied","Data":"c8bb32f9a4ece2c21df5f1129e12116611189c0133ac61102705f1471bd4abee"}
Sep 17 18:04:43.085059 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.084687    2562 scope.go:117] "RemoveContainer" containerID="a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8"
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.090196    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-rbac-proxy-sar-config\") pod \"1ebe2dd6-2e33-404b-aefd-db962812555a\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") "
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.090262    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-cabundle-cert\") pod \"1ebe2dd6-2e33-404b-aefd-db962812555a\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") "
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.090312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk592\" (UniqueName: \"kubernetes.io/projected/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-api-access-qk592\") pod \"1ebe2dd6-2e33-404b-aefd-db962812555a\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") "
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.090388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebe2dd6-2e33-404b-aefd-db962812555a-proxy-tls\") pod \"1ebe2dd6-2e33-404b-aefd-db962812555a\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") "
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.090422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2dd6-2e33-404b-aefd-db962812555a-kserve-provision-location\") pod \"1ebe2dd6-2e33-404b-aefd-db962812555a\" (UID: \"1ebe2dd6-2e33-404b-aefd-db962812555a\") "
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.091085    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebe2dd6-2e33-404b-aefd-db962812555a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ebe2dd6-2e33-404b-aefd-db962812555a" (UID: "1ebe2dd6-2e33-404b-aefd-db962812555a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.091426    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ebe2dd6-2e33-404b-aefd-db962812555a" (UID: "1ebe2dd6-2e33-404b-aefd-db962812555a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:43.093354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.091743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ebe2dd6-2e33-404b-aefd-db962812555a" (UID: "1ebe2dd6-2e33-404b-aefd-db962812555a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:43.096227 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.096199    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-api-access-qk592" (OuterVolumeSpecName: "kube-api-access-qk592") pod "1ebe2dd6-2e33-404b-aefd-db962812555a" (UID: "1ebe2dd6-2e33-404b-aefd-db962812555a"). InnerVolumeSpecName "kube-api-access-qk592". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:43.097998 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.097974    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebe2dd6-2e33-404b-aefd-db962812555a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ebe2dd6-2e33-404b-aefd-db962812555a" (UID: "1ebe2dd6-2e33-404b-aefd-db962812555a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:43.098718 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.098695    2562 scope.go:117] "RemoveContainer" containerID="bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819"
Sep 17 18:04:43.110466 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.110449    2562 scope.go:117] "RemoveContainer" containerID="8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d"
Sep 17 18:04:43.118201 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.118186    2562 scope.go:117] "RemoveContainer" containerID="a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8"
Sep 17 18:04:43.118529 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:43.118509    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8\": container with ID starting with a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8 not found: ID does not exist" containerID="a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8"
Sep 17 18:04:43.118620 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.118535    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8"} err="failed to get container status \"a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8\": rpc error: code = NotFound desc = could not find container \"a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8\": container with ID starting with a4445b3ce7cab6adea249ea4add925fbf99a783e56de44d657c39a5c1d7285c8 not found: ID does not exist"
Sep 17 18:04:43.118620 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.118569    2562 scope.go:117] "RemoveContainer" containerID="bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819"
Sep 17 18:04:43.118848 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:43.118834    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819\": container with ID starting with bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819 not found: ID does not exist" containerID="bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819"
Sep 17 18:04:43.118891 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.118850    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819"} err="failed to get container status \"bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819\": rpc error: code = NotFound desc = could not find container \"bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819\": container with ID starting with bfcd65b8987823ee865a982fc0a6a013ceec0157053ef8212f15be2613502819 not found: ID does not exist"
Sep 17 18:04:43.118891 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.118866    2562 scope.go:117] "RemoveContainer" containerID="8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d"
Sep 17 18:04:43.119093 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:43.119072    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d\": container with ID starting with 8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d not found: ID does not exist" containerID="8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d"
Sep 17 18:04:43.119134 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.119097    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d"} err="failed to get container status \"8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d\": rpc error: code = NotFound desc = could not find container \"8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d\": container with ID starting with 8cd393ae686f136f7e6542db8d67888b3eb4596e5382c606feb6fcb13276142d not found: ID does not exist"
Sep 17 18:04:43.192124 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.192092    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.192124 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.192118    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebe2dd6-2e33-404b-aefd-db962812555a-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.192124 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.192127    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk592\" (UniqueName: \"kubernetes.io/projected/1ebe2dd6-2e33-404b-aefd-db962812555a-kube-api-access-qk592\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.192367 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.192136    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebe2dd6-2e33-404b-aefd-db962812555a-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.192367 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.192146    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebe2dd6-2e33-404b-aefd-db962812555a-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.413762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.413730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"]
Sep 17 18:04:43.416729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.416705    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-v4kd2"]
Sep 17 18:04:43.534146 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534110    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"]
Sep 17 18:04:43.534530 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534504    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="storage-initializer"
Sep 17 18:04:43.534530 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534532    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="storage-initializer"
Sep 17 18:04:43.534530 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534540    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="storage-initializer"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534546    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="storage-initializer"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534557    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kube-rbac-proxy"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534563    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kube-rbac-proxy"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534571    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kube-rbac-proxy"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534576    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kube-rbac-proxy"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534593    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kserve-container"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534598    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kserve-container"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534614    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container"
Sep 17 18:04:43.534658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534619    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container"
Sep 17 18:04:43.534943 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534693    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kube-rbac-proxy"
Sep 17 18:04:43.534943 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534705    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" containerName="kserve-container"
Sep 17 18:04:43.534943 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534713    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kserve-container"
Sep 17 18:04:43.534943 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.534721    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="36806458-a53b-4cde-9b99-daa98004b58d" containerName="kube-rbac-proxy"
Sep 17 18:04:43.539539 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.539511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.542749 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.542719    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:04:43.543200 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.543183    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:04:43.555832 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.555808    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"]
Sep 17 18:04:43.595384 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.595352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.595532 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.595402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4f2\" (UniqueName: \"kubernetes.io/projected/74f493f5-839c-404a-9193-7f9546e4a180-kube-api-access-fz4f2\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.595532 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.595439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f493f5-839c-404a-9193-7f9546e4a180-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.595532 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.595466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.595532 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.595521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.696519 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.696436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.696519 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.696484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.696519 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.696509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4f2\" (UniqueName: \"kubernetes.io/projected/74f493f5-839c-404a-9193-7f9546e4a180-kube-api-access-fz4f2\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.696813 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.696537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f493f5-839c-404a-9193-7f9546e4a180-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.696813 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.696559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.696813 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:43.696609    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:04:43.696813 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:04:43.696687    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls podName:74f493f5-839c-404a-9193-7f9546e4a180 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:44.196665764 +0000 UTC m=+1828.644496200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls") pod "isvc-canary-promote-predictor-7cf78f884f-qj9p7" (UID: "74f493f5-839c-404a-9193-7f9546e4a180") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:04:43.697043 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.696976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f493f5-839c-404a-9193-7f9546e4a180-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.697242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.697221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.697318 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.697283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.710127 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.710103    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4f2\" (UniqueName: \"kubernetes.io/projected/74f493f5-839c-404a-9193-7f9546e4a180-kube-api-access-fz4f2\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:43.727830 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:43.727804    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebe2dd6-2e33-404b-aefd-db962812555a" path="/var/lib/kubelet/pods/1ebe2dd6-2e33-404b-aefd-db962812555a/volumes"
Sep 17 18:04:44.201282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:44.201247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:44.203916 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:44.203884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-qj9p7\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:44.449781 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:44.449748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:44.728447 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:44.728369    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"]
Sep 17 18:04:44.732341 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:04:44.732299    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f493f5_839c_404a_9193_7f9546e4a180.slice/crio-d50f278eba4f4f9c83d53d748990d85294f78ee2796f6d13a0900cd9765a8c43 WatchSource:0}: Error finding container d50f278eba4f4f9c83d53d748990d85294f78ee2796f6d13a0900cd9765a8c43: Status 404 returned error can't find the container with id d50f278eba4f4f9c83d53d748990d85294f78ee2796f6d13a0900cd9765a8c43
Sep 17 18:04:45.097318 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:45.097288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerStarted","Data":"62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301"}
Sep 17 18:04:45.097506 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:45.097362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerStarted","Data":"d50f278eba4f4f9c83d53d748990d85294f78ee2796f6d13a0900cd9765a8c43"}
Sep 17 18:04:47.083843 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:47.083408    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:04:47.084266 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:47.084113    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:04:49.118984 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:49.118952    2562 generic.go:356] "Generic (PLEG): container finished" podID="74f493f5-839c-404a-9193-7f9546e4a180" containerID="62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301" exitCode=0
Sep 17 18:04:49.119342 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:49.119011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerDied","Data":"62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301"}
Sep 17 18:04:50.127205 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:50.127168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerStarted","Data":"c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4"}
Sep 17 18:04:50.127604 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:50.127235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerStarted","Data":"7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137"}
Sep 17 18:04:50.127604 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:50.127424    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:50.152182 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:50.152132    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podStartSLOduration=3.389339684 podStartE2EDuration="7.152114723s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.762775039s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:50.150493884 +0000 UTC m=+1834.598324378" watchObservedRunningTime="2026-09-17 18:04:50.152114723 +0000 UTC m=+1834.599945179"
Sep 17 18:04:51.131974 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:51.131943    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:51.133309 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:51.133270    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:04:52.136191 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:52.136151    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:04:57.084808 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:57.084767    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:04:57.140502 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:57.140475    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:04:57.141001 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:04:57.140975    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:05:07.085153 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:07.085097    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:05:07.141103 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:07.141053    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:05:17.085000 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:17.084964    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:05:17.141547 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:17.141511    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:05:27.084789 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:27.084751    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:05:27.141418 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:27.141387    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:05:37.084782 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:37.084694    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:05:37.141755 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:37.141721    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:05:47.084472 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:47.084416    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:05:47.141762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:47.141725    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:05:57.085348 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:57.085303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:05:57.140952 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:05:57.140909    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:06:05.008983 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:05.008949    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"]
Sep 17 18:06:05.009528 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:05.009184    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" containerID="cri-o://13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683" gracePeriod=30
Sep 17 18:06:05.009528 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:05.009237    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kube-rbac-proxy" containerID="cri-o://f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542" gracePeriod=30
Sep 17 18:06:05.479708 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:05.479679    2562 generic.go:356] "Generic (PLEG): container finished" podID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerID="f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542" exitCode=2
Sep 17 18:06:05.479890 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:05.479755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerDied","Data":"f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542"}
Sep 17 18:06:07.079445 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:07.079399    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 18:06:07.085034 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:07.085004    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:06:07.141494 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:07.141466    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:06:09.860708 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.860684    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:06:09.944961 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.944876    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-cabundle-cert\") pod \"5d5a47aa-3e4d-47ee-9940-69e118659de0\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") "
Sep 17 18:06:09.945125 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.944971    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d5a47aa-3e4d-47ee-9940-69e118659de0-proxy-tls\") pod \"5d5a47aa-3e4d-47ee-9940-69e118659de0\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") "
Sep 17 18:06:09.945125 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxqc9\" (UniqueName: \"kubernetes.io/projected/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-api-access-gxqc9\") pod \"5d5a47aa-3e4d-47ee-9940-69e118659de0\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") "
Sep 17 18:06:09.945125 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5a47aa-3e4d-47ee-9940-69e118659de0-kserve-provision-location\") pod \"5d5a47aa-3e4d-47ee-9940-69e118659de0\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") "
Sep 17 18:06:09.945296 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-rbac-proxy-sar-config\") pod \"5d5a47aa-3e4d-47ee-9940-69e118659de0\" (UID: \"5d5a47aa-3e4d-47ee-9940-69e118659de0\") "
Sep 17 18:06:09.945411 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5a47aa-3e4d-47ee-9940-69e118659de0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d5a47aa-3e4d-47ee-9940-69e118659de0" (UID: "5d5a47aa-3e4d-47ee-9940-69e118659de0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:09.945472 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945440    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d5a47aa-3e4d-47ee-9940-69e118659de0" (UID: "5d5a47aa-3e4d-47ee-9940-69e118659de0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:09.945512 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945485    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5a47aa-3e4d-47ee-9940-69e118659de0-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:09.945548 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.945508    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d5a47aa-3e4d-47ee-9940-69e118659de0" (UID: "5d5a47aa-3e4d-47ee-9940-69e118659de0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:09.947158 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.947137    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5a47aa-3e4d-47ee-9940-69e118659de0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d5a47aa-3e4d-47ee-9940-69e118659de0" (UID: "5d5a47aa-3e4d-47ee-9940-69e118659de0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:09.947229 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:09.947177    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-api-access-gxqc9" (OuterVolumeSpecName: "kube-api-access-gxqc9") pod "5d5a47aa-3e4d-47ee-9940-69e118659de0" (UID: "5d5a47aa-3e4d-47ee-9940-69e118659de0"). InnerVolumeSpecName "kube-api-access-gxqc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:10.046288 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.046253    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d5a47aa-3e4d-47ee-9940-69e118659de0-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:10.046288 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.046289    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqc9\" (UniqueName: \"kubernetes.io/projected/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-api-access-gxqc9\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:10.046504 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.046303    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:10.046504 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.046317    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5a47aa-3e4d-47ee-9940-69e118659de0-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:10.120422 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120384    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"]
Sep 17 18:06:10.120756 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120743    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kube-rbac-proxy"
Sep 17 18:06:10.120812 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120758    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kube-rbac-proxy"
Sep 17 18:06:10.120812 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120775    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="storage-initializer"
Sep 17 18:06:10.120812 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120783    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="storage-initializer"
Sep 17 18:06:10.120812 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120793    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container"
Sep 17 18:06:10.120812 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120800    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container"
Sep 17 18:06:10.120964 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120886    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kserve-container"
Sep 17 18:06:10.120964 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.120896    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerName="kube-rbac-proxy"
Sep 17 18:06:10.124294 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.124276    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.128385 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.128362    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 18:06:10.128542 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.128522    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 18:06:10.141387 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.141362    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"]
Sep 17 18:06:10.146737 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.146715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fbf5d06-80f8-4b16-a366-31ba08b49984-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.146851 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.146760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgqt\" (UniqueName: \"kubernetes.io/projected/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-api-access-9hgqt\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.146907 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.146859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.146907 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.146895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.147018 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.146979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248112 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248112 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248060    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fbf5d06-80f8-4b16-a366-31ba08b49984-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248112 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgqt\" (UniqueName: \"kubernetes.io/projected/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-api-access-9hgqt\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248444 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248444 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248444 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:10.248208    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 18:06:10.248444 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:10.248285    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls podName:0fbf5d06-80f8-4b16-a366-31ba08b49984 nodeName:}" failed. No retries permitted until 2026-09-17 18:06:10.748263039 +0000 UTC m=+1915.196093489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" (UID: "0fbf5d06-80f8-4b16-a366-31ba08b49984") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 17 18:06:10.248616 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fbf5d06-80f8-4b16-a366-31ba08b49984-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.248839 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.249007 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.248985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.261237 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.261213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgqt\" (UniqueName: \"kubernetes.io/projected/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-api-access-9hgqt\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.505268 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.505182    2562 generic.go:356] "Generic (PLEG): container finished" podID="5d5a47aa-3e4d-47ee-9940-69e118659de0" containerID="13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683" exitCode=0
Sep 17 18:06:10.505268 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.505247    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"
Sep 17 18:06:10.505268 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.505246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerDied","Data":"13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683"}
Sep 17 18:06:10.505586 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.505323    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s" event={"ID":"5d5a47aa-3e4d-47ee-9940-69e118659de0","Type":"ContainerDied","Data":"a11644ad0b50a32b88710202264071c034a349c7327dc8f2f3195ae6326b20ca"}
Sep 17 18:06:10.505586 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.505365    2562 scope.go:117] "RemoveContainer" containerID="f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542"
Sep 17 18:06:10.514416 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.514399    2562 scope.go:117] "RemoveContainer" containerID="13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683"
Sep 17 18:06:10.521977 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.521960    2562 scope.go:117] "RemoveContainer" containerID="20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576"
Sep 17 18:06:10.529082 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.529060    2562 scope.go:117] "RemoveContainer" containerID="f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542"
Sep 17 18:06:10.529388 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:10.529362    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542\": container with ID starting with f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542 not found: ID does not exist" containerID="f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542"
Sep 17 18:06:10.529494 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.529400    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542"} err="failed to get container status \"f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542\": rpc error: code = NotFound desc = could not find container \"f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542\": container with ID starting with f91dcad60bed057084b95813a2e8d14caabb07bd4127275f30e3ddb1a591b542 not found: ID does not exist"
Sep 17 18:06:10.529494 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.529426    2562 scope.go:117] "RemoveContainer" containerID="13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683"
Sep 17 18:06:10.529680 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:10.529661    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683\": container with ID starting with 13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683 not found: ID does not exist" containerID="13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683"
Sep 17 18:06:10.529749 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.529689    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683"} err="failed to get container status \"13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683\": rpc error: code = NotFound desc = could not find container \"13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683\": container with ID starting with 13af252c62214d8690c8ede4ff42dd48aea38ff8838f75c74514f6946e980683 not found: ID does not exist"
Sep 17 18:06:10.529749 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.529715    2562 scope.go:117] "RemoveContainer" containerID="20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576"
Sep 17 18:06:10.530008 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:10.529990    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576\": container with ID starting with 20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576 not found: ID does not exist" containerID="20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576"
Sep 17 18:06:10.530060 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.530013    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576"} err="failed to get container status \"20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576\": rpc error: code = NotFound desc = could not find container \"20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576\": container with ID starting with 20d6fe46f4acd2881f142ed6e7d44dcc65c437f0e2123dc6605d3d819c526576 not found: ID does not exist"
Sep 17 18:06:10.538804 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.538773    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"]
Sep 17 18:06:10.546527 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.546506    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ffc44684-gns9s"]
Sep 17 18:06:10.752275 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.752235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:10.754807 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:10.754785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:11.034738 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.034694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:11.251152 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.251118    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"]
Sep 17 18:06:11.251389 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.251359    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container" containerID="cri-o://7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137" gracePeriod=30
Sep 17 18:06:11.251463 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.251394    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kube-rbac-proxy" containerID="cri-o://c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4" gracePeriod=30
Sep 17 18:06:11.319443 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.319409    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"]
Sep 17 18:06:11.322439 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:06:11.322411    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbf5d06_80f8_4b16_a366_31ba08b49984.slice/crio-c540643e77e1156eb83e3e2c8b72810e5cd0618867c912246ef166b2afb518a8 WatchSource:0}: Error finding container c540643e77e1156eb83e3e2c8b72810e5cd0618867c912246ef166b2afb518a8: Status 404 returned error can't find the container with id c540643e77e1156eb83e3e2c8b72810e5cd0618867c912246ef166b2afb518a8
Sep 17 18:06:11.512915 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.512868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerStarted","Data":"a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964"}
Sep 17 18:06:11.513105 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.512954    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerStarted","Data":"c540643e77e1156eb83e3e2c8b72810e5cd0618867c912246ef166b2afb518a8"}
Sep 17 18:06:11.516234 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.516206    2562 generic.go:356] "Generic (PLEG): container finished" podID="74f493f5-839c-404a-9193-7f9546e4a180" containerID="c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4" exitCode=2
Sep 17 18:06:11.516387 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.516274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerDied","Data":"c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4"}
Sep 17 18:06:11.727245 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:11.727163    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5a47aa-3e4d-47ee-9940-69e118659de0" path="/var/lib/kubelet/pods/5d5a47aa-3e4d-47ee-9940-69e118659de0/volumes"
Sep 17 18:06:12.137058 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:12.137011    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 18:06:13.568442 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.568409    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"]
Sep 17 18:06:13.571902 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.571883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.574985 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.574966    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:06:13.588360 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.588318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"]
Sep 17 18:06:13.679839 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.679798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsn9z\" (UniqueName: \"kubernetes.io/projected/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-api-access-fsn9z\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.680024 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.679855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.680024 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.679918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.680024 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.679980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb71d268-ba92-468d-93ce-07bf03a6b66c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.680024 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.680001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb71d268-ba92-468d-93ce-07bf03a6b66c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.780923 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.780884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb71d268-ba92-468d-93ce-07bf03a6b66c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.780923 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.780927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb71d268-ba92-468d-93ce-07bf03a6b66c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.781149 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.780971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsn9z\" (UniqueName: \"kubernetes.io/projected/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-api-access-fsn9z\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.781149 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.781012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.781149 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.781056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.781438 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.781405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb71d268-ba92-468d-93ce-07bf03a6b66c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.781787 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.781765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.781787 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.781778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.783479 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.783460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb71d268-ba92-468d-93ce-07bf03a6b66c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.791257 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.791232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsn9z\" (UniqueName: \"kubernetes.io/projected/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-api-access-fsn9z\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-dt75f\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:13.881777 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:13.881692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:14.158674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:14.158603    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"]
Sep 17 18:06:14.531966 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:14.531882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerStarted","Data":"5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1"}
Sep 17 18:06:14.531966 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:14.531941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerStarted","Data":"df13df4adb20e283b9a9bf7c7078fa4335de03c0765a7a4b27b8bb8bad9d7bc0"}
Sep 17 18:06:15.537850 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:15.537818    2562 generic.go:356] "Generic (PLEG): container finished" podID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerID="a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964" exitCode=0
Sep 17 18:06:15.538253 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:15.537891    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerDied","Data":"a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964"}
Sep 17 18:06:16.293912 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.293882    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:06:16.405698 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.405618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-kube-rbac-proxy-sar-config\") pod \"74f493f5-839c-404a-9193-7f9546e4a180\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") "
Sep 17 18:06:16.405698 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.405653    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls\") pod \"74f493f5-839c-404a-9193-7f9546e4a180\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") "
Sep 17 18:06:16.405871 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.405725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f493f5-839c-404a-9193-7f9546e4a180-kserve-provision-location\") pod \"74f493f5-839c-404a-9193-7f9546e4a180\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") "
Sep 17 18:06:16.405916 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.405864    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz4f2\" (UniqueName: \"kubernetes.io/projected/74f493f5-839c-404a-9193-7f9546e4a180-kube-api-access-fz4f2\") pod \"74f493f5-839c-404a-9193-7f9546e4a180\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") "
Sep 17 18:06:16.406042 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.405912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-cabundle-cert\") pod \"74f493f5-839c-404a-9193-7f9546e4a180\" (UID: \"74f493f5-839c-404a-9193-7f9546e4a180\") "
Sep 17 18:06:16.406109 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.406044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74f493f5-839c-404a-9193-7f9546e4a180" (UID: "74f493f5-839c-404a-9193-7f9546e4a180"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:16.406109 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.406078    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f493f5-839c-404a-9193-7f9546e4a180-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74f493f5-839c-404a-9193-7f9546e4a180" (UID: "74f493f5-839c-404a-9193-7f9546e4a180"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:16.406219 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.406147    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74f493f5-839c-404a-9193-7f9546e4a180-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:16.406219 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.406170    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:16.406313 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.406242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74f493f5-839c-404a-9193-7f9546e4a180" (UID: "74f493f5-839c-404a-9193-7f9546e4a180"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:16.407890 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.407866    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74f493f5-839c-404a-9193-7f9546e4a180" (UID: "74f493f5-839c-404a-9193-7f9546e4a180"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:16.407981 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.407954    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f493f5-839c-404a-9193-7f9546e4a180-kube-api-access-fz4f2" (OuterVolumeSpecName: "kube-api-access-fz4f2") pod "74f493f5-839c-404a-9193-7f9546e4a180" (UID: "74f493f5-839c-404a-9193-7f9546e4a180"). InnerVolumeSpecName "kube-api-access-fz4f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:16.507544 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.507505    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74f493f5-839c-404a-9193-7f9546e4a180-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:16.507544 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.507538    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4f2\" (UniqueName: \"kubernetes.io/projected/74f493f5-839c-404a-9193-7f9546e4a180-kube-api-access-fz4f2\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:16.507544 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.507550    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74f493f5-839c-404a-9193-7f9546e4a180-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:06:16.545219 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.545186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerStarted","Data":"731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee"}
Sep 17 18:06:16.545639 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.545254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerStarted","Data":"6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb"}
Sep 17 18:06:16.545639 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.545474    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:16.546954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.546928    2562 generic.go:356] "Generic (PLEG): container finished" podID="74f493f5-839c-404a-9193-7f9546e4a180" containerID="7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137" exitCode=0
Sep 17 18:06:16.547038 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.546981    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"
Sep 17 18:06:16.547038 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.547011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerDied","Data":"7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137"}
Sep 17 18:06:16.547116 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.547065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7" event={"ID":"74f493f5-839c-404a-9193-7f9546e4a180","Type":"ContainerDied","Data":"d50f278eba4f4f9c83d53d748990d85294f78ee2796f6d13a0900cd9765a8c43"}
Sep 17 18:06:16.547116 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.547092    2562 scope.go:117] "RemoveContainer" containerID="c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4"
Sep 17 18:06:16.555908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.555890    2562 scope.go:117] "RemoveContainer" containerID="7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137"
Sep 17 18:06:16.563074 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.563059    2562 scope.go:117] "RemoveContainer" containerID="62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301"
Sep 17 18:06:16.570371 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.570310    2562 scope.go:117] "RemoveContainer" containerID="c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4"
Sep 17 18:06:16.570655 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:16.570628    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4\": container with ID starting with c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4 not found: ID does not exist" containerID="c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4"
Sep 17 18:06:16.570771 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.570667    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4"} err="failed to get container status \"c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4\": rpc error: code = NotFound desc = could not find container \"c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4\": container with ID starting with c991eaae75f0bde7f38aec7fde6811ff449f2d4e220b5ecf23681dd21ba913d4 not found: ID does not exist"
Sep 17 18:06:16.570771 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.570691    2562 scope.go:117] "RemoveContainer" containerID="7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137"
Sep 17 18:06:16.570966 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:16.570946    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137\": container with ID starting with 7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137 not found: ID does not exist" containerID="7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137"
Sep 17 18:06:16.571035 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.570977    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137"} err="failed to get container status \"7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137\": rpc error: code = NotFound desc = could not find container \"7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137\": container with ID starting with 7c7161eac2416efbc6aa4cd841873721c6ec63e100b62e854bf49ee580a80137 not found: ID does not exist"
Sep 17 18:06:16.571035 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.570991    2562 scope.go:117] "RemoveContainer" containerID="62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301"
Sep 17 18:06:16.571255 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:06:16.571236    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301\": container with ID starting with 62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301 not found: ID does not exist" containerID="62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301"
Sep 17 18:06:16.571314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.571265    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301"} err="failed to get container status \"62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301\": rpc error: code = NotFound desc = could not find container \"62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301\": container with ID starting with 62380d65af276bc093a6b9bb2dc1b0d0b42d8052251f38cfe5d1202be87f2301 not found: ID does not exist"
Sep 17 18:06:16.571614 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.571580    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podStartSLOduration=2.5848737010000002 podStartE2EDuration="6.571568644s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986694943s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:16.57014167 +0000 UTC m=+1921.017972188" watchObservedRunningTime="2026-09-17 18:06:16.571568644 +0000 UTC m=+1921.019399099"
Sep 17 18:06:16.587011 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.586989    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"]
Sep 17 18:06:16.588669 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:16.588651    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-qj9p7"]
Sep 17 18:06:17.552613 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:17.552578    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:17.553905 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:17.553873    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:06:17.732731 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:17.732654    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f493f5-839c-404a-9193-7f9546e4a180" path="/var/lib/kubelet/pods/74f493f5-839c-404a-9193-7f9546e4a180/volumes"
Sep 17 18:06:18.557954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:18.557928    2562 generic.go:356] "Generic (PLEG): container finished" podID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerID="5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1" exitCode=0
Sep 17 18:06:18.558314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:18.557996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerDied","Data":"5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1"}
Sep 17 18:06:18.558403 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:18.558312    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:06:19.566002 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:19.565964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerStarted","Data":"870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c"}
Sep 17 18:06:19.566386 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:19.566033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerStarted","Data":"166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09"}
Sep 17 18:06:19.566386 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:19.566360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:19.591354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:19.591299    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podStartSLOduration=2.821422594 podStartE2EDuration="6.591285755s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.769863161s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:19.588934536 +0000 UTC m=+1924.036764990" watchObservedRunningTime="2026-09-17 18:06:19.591285755 +0000 UTC m=+1924.039116209"
Sep 17 18:06:20.570786 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:20.570755    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:20.572152 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:20.572124    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:06:21.575537 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:21.575498    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:06:23.563056 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:23.563025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:06:23.563728 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:23.563696    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:06:26.580390 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:26.580361    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:06:26.580910 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:26.580886    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:06:33.564363 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:33.564306    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:06:36.581190 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:36.581145    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:06:43.566857 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:43.563923    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:06:46.581825 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:46.581788    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:06:53.564302 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:53.564262    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:06:56.580849 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:06:56.580804    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:07:03.563876 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:03.563792    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:07:06.581171 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:06.581133    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:07:13.563952 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:13.563912    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:07:16.581158 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:16.581114    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:07:23.563834 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:23.563791    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:07:26.581676 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:26.581638    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:07:33.564515 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:33.564481    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:07:36.581418 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:36.581390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:07:40.303225 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:40.303190    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"]
Sep 17 18:07:40.303637 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:40.303447    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" containerID="cri-o://6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb" gracePeriod=30
Sep 17 18:07:40.303637 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:40.303471    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kube-rbac-proxy" containerID="cri-o://731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee" gracePeriod=30
Sep 17 18:07:40.962535 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:40.962501    2562 generic.go:356] "Generic (PLEG): container finished" podID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerID="731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee" exitCode=2
Sep 17 18:07:40.962696 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:40.962570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerDied","Data":"731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee"}
Sep 17 18:07:43.558919 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:43.558868    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 18:07:43.564314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:43.564281    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:07:43.867764 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:43.867733    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"]
Sep 17 18:07:43.868012 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:43.867984    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kube-rbac-proxy" containerID="cri-o://870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c" gracePeriod=30
Sep 17 18:07:43.868084 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:43.867966    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" containerID="cri-o://166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09" gracePeriod=30
Sep 17 18:07:44.852085 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.852059    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:07:44.961647 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961554    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-cabundle-cert\") pod \"0fbf5d06-80f8-4b16-a366-31ba08b49984\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") "
Sep 17 18:07:44.961647 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fbf5d06-80f8-4b16-a366-31ba08b49984-kserve-provision-location\") pod \"0fbf5d06-80f8-4b16-a366-31ba08b49984\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") "
Sep 17 18:07:44.961647 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-rbac-proxy-sar-config\") pod \"0fbf5d06-80f8-4b16-a366-31ba08b49984\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") "
Sep 17 18:07:44.961941 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961669    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hgqt\" (UniqueName: \"kubernetes.io/projected/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-api-access-9hgqt\") pod \"0fbf5d06-80f8-4b16-a366-31ba08b49984\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") "
Sep 17 18:07:44.961941 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961703    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls\") pod \"0fbf5d06-80f8-4b16-a366-31ba08b49984\" (UID: \"0fbf5d06-80f8-4b16-a366-31ba08b49984\") "
Sep 17 18:07:44.962043 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961953    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbf5d06-80f8-4b16-a366-31ba08b49984-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fbf5d06-80f8-4b16-a366-31ba08b49984" (UID: "0fbf5d06-80f8-4b16-a366-31ba08b49984"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:44.962043 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.961981    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0fbf5d06-80f8-4b16-a366-31ba08b49984" (UID: "0fbf5d06-80f8-4b16-a366-31ba08b49984"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:44.962156 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.962058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0fbf5d06-80f8-4b16-a366-31ba08b49984" (UID: "0fbf5d06-80f8-4b16-a366-31ba08b49984"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:44.963725 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.963704    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0fbf5d06-80f8-4b16-a366-31ba08b49984" (UID: "0fbf5d06-80f8-4b16-a366-31ba08b49984"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:44.963859 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.963843    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-api-access-9hgqt" (OuterVolumeSpecName: "kube-api-access-9hgqt") pod "0fbf5d06-80f8-4b16-a366-31ba08b49984" (UID: "0fbf5d06-80f8-4b16-a366-31ba08b49984"). InnerVolumeSpecName "kube-api-access-9hgqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:44.981793 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.981768    2562 generic.go:356] "Generic (PLEG): container finished" podID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerID="6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb" exitCode=0
Sep 17 18:07:44.981929 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.981822    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"
Sep 17 18:07:44.981929 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.981838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerDied","Data":"6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb"}
Sep 17 18:07:44.981929 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.981893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj" event={"ID":"0fbf5d06-80f8-4b16-a366-31ba08b49984","Type":"ContainerDied","Data":"c540643e77e1156eb83e3e2c8b72810e5cd0618867c912246ef166b2afb518a8"}
Sep 17 18:07:44.981929 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.981928    2562 scope.go:117] "RemoveContainer" containerID="731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee"
Sep 17 18:07:44.984422 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.984392    2562 generic.go:356] "Generic (PLEG): container finished" podID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerID="870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c" exitCode=2
Sep 17 18:07:44.984517 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.984453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerDied","Data":"870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c"}
Sep 17 18:07:44.990495 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.990475    2562 scope.go:117] "RemoveContainer" containerID="6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb"
Sep 17 18:07:44.997873 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:44.997859    2562 scope.go:117] "RemoveContainer" containerID="a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964"
Sep 17 18:07:45.004724 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.004708    2562 scope.go:117] "RemoveContainer" containerID="731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee"
Sep 17 18:07:45.004980 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:45.004965    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee\": container with ID starting with 731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee not found: ID does not exist" containerID="731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee"
Sep 17 18:07:45.005063 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.004991    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee"} err="failed to get container status \"731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee\": rpc error: code = NotFound desc = could not find container \"731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee\": container with ID starting with 731e715e3daeb42f60cb4f302f65bc5d156bf175378e7891f979c47e3156b4ee not found: ID does not exist"
Sep 17 18:07:45.005063 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.005013    2562 scope.go:117] "RemoveContainer" containerID="6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb"
Sep 17 18:07:45.005272 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:45.005253    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb\": container with ID starting with 6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb not found: ID does not exist" containerID="6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb"
Sep 17 18:07:45.005312 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.005278    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb"} err="failed to get container status \"6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb\": rpc error: code = NotFound desc = could not find container \"6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb\": container with ID starting with 6f53e8b21c807207e465b53f253b157e687ffabfdd5972b4874b4e23ec076deb not found: ID does not exist"
Sep 17 18:07:45.005312 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.005293    2562 scope.go:117] "RemoveContainer" containerID="a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964"
Sep 17 18:07:45.005521 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:45.005506    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964\": container with ID starting with a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964 not found: ID does not exist" containerID="a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964"
Sep 17 18:07:45.005572 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.005529    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964"} err="failed to get container status \"a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964\": rpc error: code = NotFound desc = could not find container \"a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964\": container with ID starting with a62f11d5f320b3b39a5561c95f4e9d830b31637039a02fef02733c87d8745964 not found: ID does not exist"
Sep 17 18:07:45.020264 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.020244    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"]
Sep 17 18:07:45.021918 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.021888    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-89c454589-dtgkj"]
Sep 17 18:07:45.062863 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.062838    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0fbf5d06-80f8-4b16-a366-31ba08b49984-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:45.062863 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.062861    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:45.063005 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.062876    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fbf5d06-80f8-4b16-a366-31ba08b49984-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:45.063005 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.062889    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:45.063005 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.062901    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hgqt\" (UniqueName: \"kubernetes.io/projected/0fbf5d06-80f8-4b16-a366-31ba08b49984-kube-api-access-9hgqt\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:45.727839 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.727808    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" path="/var/lib/kubelet/pods/0fbf5d06-80f8-4b16-a366-31ba08b49984/volumes"
Sep 17 18:07:45.735071 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735046    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"]
Sep 17 18:07:45.735422 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735408    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container"
Sep 17 18:07:45.735422 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735423    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735430    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kube-rbac-proxy"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735436    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kube-rbac-proxy"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735445    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kube-rbac-proxy"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735451    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kube-rbac-proxy"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735464    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="storage-initializer"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735470    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="storage-initializer"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735478    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="storage-initializer"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735483    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="storage-initializer"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735492    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container"
Sep 17 18:07:45.735509 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735497    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container"
Sep 17 18:07:45.735802 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735551    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kserve-container"
Sep 17 18:07:45.735802 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735560    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kserve-container"
Sep 17 18:07:45.735802 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735567    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbf5d06-80f8-4b16-a366-31ba08b49984" containerName="kube-rbac-proxy"
Sep 17 18:07:45.735802 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.735572    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f493f5-839c-404a-9193-7f9546e4a180" containerName="kube-rbac-proxy"
Sep 17 18:07:45.740265 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.740248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.743746 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.743727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 18:07:45.744014 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.743995    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 18:07:45.769100 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.769071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39bddd66-eec1-4f05-960b-ee43d1358d9a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.769210 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.769115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.769276 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.769252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.769367 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.769304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.769421 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.769366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fb9d\" (UniqueName: \"kubernetes.io/projected/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-api-access-6fb9d\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.775740 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.775715    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"]
Sep 17 18:07:45.870091 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.870551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.870551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fb9d\" (UniqueName: \"kubernetes.io/projected/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-api-access-6fb9d\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.870551 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:45.870225    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:07:45.870551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39bddd66-eec1-4f05-960b-ee43d1358d9a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.870551 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:45.870364    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls podName:39bddd66-eec1-4f05-960b-ee43d1358d9a nodeName:}" failed. No retries permitted until 2026-09-17 18:07:46.370317043 +0000 UTC m=+2010.818147493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-5x2gq" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:07:45.870551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.870954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39bddd66-eec1-4f05-960b-ee43d1358d9a-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.870954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.871034 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.870951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:45.887980 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:45.887954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fb9d\" (UniqueName: \"kubernetes.io/projected/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-api-access-6fb9d\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:46.375126 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:46.375090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:46.375350 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:46.375263    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:07:46.375419 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:46.375364    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls podName:39bddd66-eec1-4f05-960b-ee43d1358d9a nodeName:}" failed. No retries permitted until 2026-09-17 18:07:47.375315737 +0000 UTC m=+2011.823146175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-5x2gq" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 18:07:46.576109 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:46.576064    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 18:07:46.581551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:46.581521    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:07:47.383373 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:47.383317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:47.385747 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:47.385716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-5x2gq\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:47.550267 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:47.550231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:07:47.839593 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:47.839512    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"]
Sep 17 18:07:47.842544 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:07:47.842519    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bddd66_eec1_4f05_960b_ee43d1358d9a.slice/crio-37dd0cf1f59c8390125fad202870bc9fb47074f832d25886978d39963008ba98 WatchSource:0}: Error finding container 37dd0cf1f59c8390125fad202870bc9fb47074f832d25886978d39963008ba98: Status 404 returned error can't find the container with id 37dd0cf1f59c8390125fad202870bc9fb47074f832d25886978d39963008ba98
Sep 17 18:07:48.002858 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.002819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerStarted","Data":"9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6"}
Sep 17 18:07:48.003055 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.002882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerStarted","Data":"37dd0cf1f59c8390125fad202870bc9fb47074f832d25886978d39963008ba98"}
Sep 17 18:07:48.606712 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.606689    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:07:48.694742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.694676    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb71d268-ba92-468d-93ce-07bf03a6b66c-kserve-provision-location\") pod \"eb71d268-ba92-468d-93ce-07bf03a6b66c\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") "
Sep 17 18:07:48.694742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.694722    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-cabundle-cert\") pod \"eb71d268-ba92-468d-93ce-07bf03a6b66c\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") "
Sep 17 18:07:48.694927 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.694747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb71d268-ba92-468d-93ce-07bf03a6b66c-proxy-tls\") pod \"eb71d268-ba92-468d-93ce-07bf03a6b66c\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") "
Sep 17 18:07:48.694927 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.694808    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsn9z\" (UniqueName: \"kubernetes.io/projected/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-api-access-fsn9z\") pod \"eb71d268-ba92-468d-93ce-07bf03a6b66c\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") "
Sep 17 18:07:48.694927 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.694826    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-rbac-proxy-sar-config\") pod \"eb71d268-ba92-468d-93ce-07bf03a6b66c\" (UID: \"eb71d268-ba92-468d-93ce-07bf03a6b66c\") "
Sep 17 18:07:48.695094 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.695018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb71d268-ba92-468d-93ce-07bf03a6b66c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb71d268-ba92-468d-93ce-07bf03a6b66c" (UID: "eb71d268-ba92-468d-93ce-07bf03a6b66c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:48.695094 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.695053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb71d268-ba92-468d-93ce-07bf03a6b66c" (UID: "eb71d268-ba92-468d-93ce-07bf03a6b66c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:48.695248 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.695229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb71d268-ba92-468d-93ce-07bf03a6b66c" (UID: "eb71d268-ba92-468d-93ce-07bf03a6b66c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:48.696824 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.696798    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-api-access-fsn9z" (OuterVolumeSpecName: "kube-api-access-fsn9z") pod "eb71d268-ba92-468d-93ce-07bf03a6b66c" (UID: "eb71d268-ba92-468d-93ce-07bf03a6b66c"). InnerVolumeSpecName "kube-api-access-fsn9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:48.696824 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.696812    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb71d268-ba92-468d-93ce-07bf03a6b66c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb71d268-ba92-468d-93ce-07bf03a6b66c" (UID: "eb71d268-ba92-468d-93ce-07bf03a6b66c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:48.796212 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.796183    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.796212 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.796207    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb71d268-ba92-468d-93ce-07bf03a6b66c-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.796212 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.796216    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb71d268-ba92-468d-93ce-07bf03a6b66c-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.796436 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.796224    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb71d268-ba92-468d-93ce-07bf03a6b66c-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.796436 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:48.796233    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsn9z\" (UniqueName: \"kubernetes.io/projected/eb71d268-ba92-468d-93ce-07bf03a6b66c-kube-api-access-fsn9z\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:07:49.011191 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.011100    2562 generic.go:356] "Generic (PLEG): container finished" podID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerID="166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09" exitCode=0
Sep 17 18:07:49.011191 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.011137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerDied","Data":"166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09"}
Sep 17 18:07:49.011191 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.011165    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"
Sep 17 18:07:49.011523 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.011215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f" event={"ID":"eb71d268-ba92-468d-93ce-07bf03a6b66c","Type":"ContainerDied","Data":"df13df4adb20e283b9a9bf7c7078fa4335de03c0765a7a4b27b8bb8bad9d7bc0"}
Sep 17 18:07:49.011523 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.011253    2562 scope.go:117] "RemoveContainer" containerID="870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c"
Sep 17 18:07:49.023613 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.023591    2562 scope.go:117] "RemoveContainer" containerID="166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09"
Sep 17 18:07:49.030496 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:49.030466    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb71d268_ba92_468d_93ce_07bf03a6b66c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb71d268_ba92_468d_93ce_07bf03a6b66c.slice/crio-df13df4adb20e283b9a9bf7c7078fa4335de03c0765a7a4b27b8bb8bad9d7bc0\": RecentStats: unable to find data in memory cache]"
Sep 17 18:07:49.033514 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.033496    2562 scope.go:117] "RemoveContainer" containerID="5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1"
Sep 17 18:07:49.044152 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.044132    2562 scope.go:117] "RemoveContainer" containerID="870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c"
Sep 17 18:07:49.044474 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:49.044456    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c\": container with ID starting with 870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c not found: ID does not exist" containerID="870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c"
Sep 17 18:07:49.044537 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.044482    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c"} err="failed to get container status \"870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c\": rpc error: code = NotFound desc = could not find container \"870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c\": container with ID starting with 870cb9d07c3a22a411f8da929af39f8e836397e17472aed35fa18b0ec7009c8c not found: ID does not exist"
Sep 17 18:07:49.044537 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.044499    2562 scope.go:117] "RemoveContainer" containerID="166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09"
Sep 17 18:07:49.044767 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:49.044749    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09\": container with ID starting with 166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09 not found: ID does not exist" containerID="166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09"
Sep 17 18:07:49.044830 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.044775    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09"} err="failed to get container status \"166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09\": rpc error: code = NotFound desc = could not find container \"166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09\": container with ID starting with 166d902abd27908197d1bc6d757aecc00d2ce093f0862e4dd260a7b0c917fc09 not found: ID does not exist"
Sep 17 18:07:49.044830 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.044798    2562 scope.go:117] "RemoveContainer" containerID="5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1"
Sep 17 18:07:49.045101 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:49.045073    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1\": container with ID starting with 5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1 not found: ID does not exist" containerID="5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1"
Sep 17 18:07:49.045157 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.045096    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1"} err="failed to get container status \"5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1\": rpc error: code = NotFound desc = could not find container \"5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1\": container with ID starting with 5f62fa334acec68b404bcba870745378a023308c43533a7fa38ab4258ee536a1 not found: ID does not exist"
Sep 17 18:07:49.050473 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.050452    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"]
Sep 17 18:07:49.064054 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.064029    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-dt75f"]
Sep 17 18:07:49.444898 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.444861    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"]
Sep 17 18:07:49.445274 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445256    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container"
Sep 17 18:07:49.445372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container"
Sep 17 18:07:49.445372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445294    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="storage-initializer"
Sep 17 18:07:49.445372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445301    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="storage-initializer"
Sep 17 18:07:49.445372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445315    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kube-rbac-proxy"
Sep 17 18:07:49.445372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445322    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kube-rbac-proxy"
Sep 17 18:07:49.445636 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445402    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kube-rbac-proxy"
Sep 17 18:07:49.445636 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.445416    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" containerName="kserve-container"
Sep 17 18:07:49.450240 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.450223    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.454278 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.454258    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:07:49.454741 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.454727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:07:49.469879 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.469853    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"]
Sep 17 18:07:49.502344 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.502297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.502502 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.502365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.502502 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.502396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.502502 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.502486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqpj\" (UniqueName: \"kubernetes.io/projected/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-api-access-gjqpj\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.502621 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.502513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271b9ec7-60b8-4512-b99e-d44068cc9262-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.603758 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.603723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.603921 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.603762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.603921 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.603786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.603921 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.603847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqpj\" (UniqueName: \"kubernetes.io/projected/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-api-access-gjqpj\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.603921 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.603870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271b9ec7-60b8-4512-b99e-d44068cc9262-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.603921 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:49.603894    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:07:49.604269 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:07:49.603966    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls podName:271b9ec7-60b8-4512-b99e-d44068cc9262 nodeName:}" failed. No retries permitted until 2026-09-17 18:07:50.103944807 +0000 UTC m=+2014.551775245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-wc592" (UID: "271b9ec7-60b8-4512-b99e-d44068cc9262") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:07:49.604269 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.604249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271b9ec7-60b8-4512-b99e-d44068cc9262-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.604480 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.604462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.604518 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.604497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.619891 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.619868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqpj\" (UniqueName: \"kubernetes.io/projected/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-api-access-gjqpj\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:49.727018 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:49.726941    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb71d268-ba92-468d-93ce-07bf03a6b66c" path="/var/lib/kubelet/pods/eb71d268-ba92-468d-93ce-07bf03a6b66c/volumes"
Sep 17 18:07:50.107293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:50.107262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:50.109586 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:50.109567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-wc592\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:50.360546 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:50.360469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:50.644359 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:50.644274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"]
Sep 17 18:07:50.648394 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:07:50.648364    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b9ec7_60b8_4512_b99e_d44068cc9262.slice/crio-28e0afffe6525da2111d44448522197fd89e467e286b8373fa80530202d7b567 WatchSource:0}: Error finding container 28e0afffe6525da2111d44448522197fd89e467e286b8373fa80530202d7b567: Status 404 returned error can't find the container with id 28e0afffe6525da2111d44448522197fd89e467e286b8373fa80530202d7b567
Sep 17 18:07:51.024256 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:51.024169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerStarted","Data":"789fd3da99998d2b9f81f3a44ae4ddb272a102281a8e43564d38118eca8d0d43"}
Sep 17 18:07:51.024256 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:51.024238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerStarted","Data":"28e0afffe6525da2111d44448522197fd89e467e286b8373fa80530202d7b567"}
Sep 17 18:07:55.044855 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:55.044822    2562 generic.go:356] "Generic (PLEG): container finished" podID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerID="789fd3da99998d2b9f81f3a44ae4ddb272a102281a8e43564d38118eca8d0d43" exitCode=0
Sep 17 18:07:55.045321 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:55.044913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerDied","Data":"789fd3da99998d2b9f81f3a44ae4ddb272a102281a8e43564d38118eca8d0d43"}
Sep 17 18:07:56.052139 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:56.052090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerStarted","Data":"cf4e3962c2ca44891651059fcc468eafc11cc3e6f629a08301d21a86793f358b"}
Sep 17 18:07:56.052548 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:56.052165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerStarted","Data":"d47343c265b815a3995e68bce1eeefadf69d96af85e682bb0143447e7fd32c81"}
Sep 17 18:07:56.052548 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:56.052366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:56.083186 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:56.083137    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podStartSLOduration=3.477816682 podStartE2EDuration="7.083123473s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.605306791s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:56.082683684 +0000 UTC m=+2020.530514164" watchObservedRunningTime="2026-09-17 18:07:56.083123473 +0000 UTC m=+2020.530953926"
Sep 17 18:07:57.057557 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:57.057519    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:07:57.058950 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:57.058920    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:07:58.061629 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:07:58.061588    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:02.080884 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:02.080848    2562 generic.go:356] "Generic (PLEG): container finished" podID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerID="9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6" exitCode=0
Sep 17 18:08:02.081397 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:02.080924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerDied","Data":"9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6"}
Sep 17 18:08:02.082407 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:02.082394    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:03.066902 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:03.066875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:08:03.067751 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:03.067721    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:06.110288 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:06.110204    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerStarted","Data":"07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90"}
Sep 17 18:08:06.110288 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:06.110268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerStarted","Data":"2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5"}
Sep 17 18:08:06.110685 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:06.110604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:08:06.110685 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:06.110658    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:08:06.112146 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:06.112120    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:08:06.159314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:06.159260    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podStartSLOduration=4.033743709 podStartE2EDuration="21.159243856s" totalImagesPullingTime="3.760901595s" totalInitContainerRuntime="13.364598552s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:06.158456277 +0000 UTC m=+2030.606286753" watchObservedRunningTime="2026-09-17 18:08:06.159243856 +0000 UTC m=+2030.607074311"
Sep 17 18:08:07.115525 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:07.115485    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:08:12.120929 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:12.120898    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:08:12.121577 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:12.121544    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:08:13.067545 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:13.067508    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:22.122278 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:22.122250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:08:23.067615 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:23.067579    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:33.068365 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:33.068251    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:36.416258 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:36.416228    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"]
Sep 17 18:08:36.416681 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:36.416470    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container" containerID="cri-o://2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5" gracePeriod=30
Sep 17 18:08:36.416681 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:36.416527    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" containerID="cri-o://07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90" gracePeriod=30
Sep 17 18:08:37.116526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:37.116483    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:08:37.266290 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:37.266256    2562 generic.go:356] "Generic (PLEG): container finished" podID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerID="07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90" exitCode=2
Sep 17 18:08:37.266460 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:37.266341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerDied","Data":"07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90"}
Sep 17 18:08:42.116319 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:42.116275    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:08:43.067428 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:43.067389    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:47.115781 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:47.115740    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:08:47.116145 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:47.115878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:08:52.116412 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:52.116358    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:08:53.067468 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:53.067432    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:08:57.116540 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:08:57.116501    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:09:02.115902 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:02.115864    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 18:09:03.067801 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:03.067755    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:09:07.064510 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.064483    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:09:07.156172 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156145    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") pod \"39bddd66-eec1-4f05-960b-ee43d1358d9a\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") "
Sep 17 18:09:07.156282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156195    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39bddd66-eec1-4f05-960b-ee43d1358d9a-kserve-provision-location\") pod \"39bddd66-eec1-4f05-960b-ee43d1358d9a\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") "
Sep 17 18:09:07.156282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156216    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-rbac-proxy-sar-config\") pod \"39bddd66-eec1-4f05-960b-ee43d1358d9a\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") "
Sep 17 18:09:07.156282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-cabundle-cert\") pod \"39bddd66-eec1-4f05-960b-ee43d1358d9a\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") "
Sep 17 18:09:07.156494 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156293    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fb9d\" (UniqueName: \"kubernetes.io/projected/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-api-access-6fb9d\") pod \"39bddd66-eec1-4f05-960b-ee43d1358d9a\" (UID: \"39bddd66-eec1-4f05-960b-ee43d1358d9a\") "
Sep 17 18:09:07.156660 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39bddd66-eec1-4f05-960b-ee43d1358d9a" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:07.156660 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.156644    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39bddd66-eec1-4f05-960b-ee43d1358d9a" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:07.158273 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.158243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39bddd66-eec1-4f05-960b-ee43d1358d9a" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:07.158384 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.158286    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-api-access-6fb9d" (OuterVolumeSpecName: "kube-api-access-6fb9d") pod "39bddd66-eec1-4f05-960b-ee43d1358d9a" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a"). InnerVolumeSpecName "kube-api-access-6fb9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:07.167583 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.167562    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bddd66-eec1-4f05-960b-ee43d1358d9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39bddd66-eec1-4f05-960b-ee43d1358d9a" (UID: "39bddd66-eec1-4f05-960b-ee43d1358d9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:07.257531 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.257467    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39bddd66-eec1-4f05-960b-ee43d1358d9a-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:09:07.257531 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.257493    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39bddd66-eec1-4f05-960b-ee43d1358d9a-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:09:07.257531 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.257506    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:09:07.257531 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.257518    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39bddd66-eec1-4f05-960b-ee43d1358d9a-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:09:07.257531 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.257531    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fb9d\" (UniqueName: \"kubernetes.io/projected/39bddd66-eec1-4f05-960b-ee43d1358d9a-kube-api-access-6fb9d\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:09:07.410712 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.410679    2562 generic.go:356] "Generic (PLEG): container finished" podID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerID="2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5" exitCode=137
Sep 17 18:09:07.410877 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.410742    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"
Sep 17 18:09:07.410877 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.410752    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerDied","Data":"2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5"}
Sep 17 18:09:07.410877 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.410837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq" event={"ID":"39bddd66-eec1-4f05-960b-ee43d1358d9a","Type":"ContainerDied","Data":"37dd0cf1f59c8390125fad202870bc9fb47074f832d25886978d39963008ba98"}
Sep 17 18:09:07.410877 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.410864    2562 scope.go:117] "RemoveContainer" containerID="07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90"
Sep 17 18:09:07.420282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.420266    2562 scope.go:117] "RemoveContainer" containerID="2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5"
Sep 17 18:09:07.428046 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.428030    2562 scope.go:117] "RemoveContainer" containerID="9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6"
Sep 17 18:09:07.437934 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.437910    2562 scope.go:117] "RemoveContainer" containerID="07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90"
Sep 17 18:09:07.438708 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:09:07.438679    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90\": container with ID starting with 07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90 not found: ID does not exist" containerID="07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90"
Sep 17 18:09:07.438801 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.438720    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90"} err="failed to get container status \"07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90\": rpc error: code = NotFound desc = could not find container \"07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90\": container with ID starting with 07483c9cc285c1a8b3906fa192d9378645525b3968299cb6e90acdfa0b97cc90 not found: ID does not exist"
Sep 17 18:09:07.438801 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.438758    2562 scope.go:117] "RemoveContainer" containerID="2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5"
Sep 17 18:09:07.439259 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:09:07.439235    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5\": container with ID starting with 2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5 not found: ID does not exist" containerID="2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5"
Sep 17 18:09:07.439349 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.439267    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5"} err="failed to get container status \"2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5\": rpc error: code = NotFound desc = could not find container \"2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5\": container with ID starting with 2c3b67ccf554f8ea11cd199647b1df7a2eaf8603402b69a1f9eba412bc5d02b5 not found: ID does not exist"
Sep 17 18:09:07.439349 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.439290    2562 scope.go:117] "RemoveContainer" containerID="9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6"
Sep 17 18:09:07.439606 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:09:07.439588    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6\": container with ID starting with 9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6 not found: ID does not exist" containerID="9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6"
Sep 17 18:09:07.439663 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.439615    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6"} err="failed to get container status \"9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6\": rpc error: code = NotFound desc = could not find container \"9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6\": container with ID starting with 9ff9f363bda634eb73b05de0ddec2a3164ecace390b55858e285e71b9181c1b6 not found: ID does not exist"
Sep 17 18:09:07.439860 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.439838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"]
Sep 17 18:09:07.443659 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.443636    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-5x2gq"]
Sep 17 18:09:07.728653 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:07.728621    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" path="/var/lib/kubelet/pods/39bddd66-eec1-4f05-960b-ee43d1358d9a/volumes"
Sep 17 18:09:13.067964 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:13.067938    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:09:17.822867 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:17.822833    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:09:17.827195 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:17.827172    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:09:27.984055 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984017    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"]
Sep 17 18:09:27.984544 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy"
Sep 17 18:09:27.984589 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984547    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy"
Sep 17 18:09:27.984589 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984564    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container"
Sep 17 18:09:27.984589 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984572    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container"
Sep 17 18:09:27.984589 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984584    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="storage-initializer"
Sep 17 18:09:27.984715 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984593    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="storage-initializer"
Sep 17 18:09:27.984715 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984673    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kserve-container"
Sep 17 18:09:27.984715 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.984688    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bddd66-eec1-4f05-960b-ee43d1358d9a" containerName="kube-rbac-proxy"
Sep 17 18:09:27.989471 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.989453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:27.992012 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.991991    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 18:09:27.992299 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:27.992285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 18:09:28.003821 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.003801    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"]
Sep 17 18:09:28.124156 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.124127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.124156 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.124155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrt9\" (UniqueName: \"kubernetes.io/projected/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-api-access-2qrt9\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.124419 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.124199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.124419 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.124241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.124419 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.124306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776e97ad-71dd-4a63-8f01-a0b856a4614c-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776e97ad-71dd-4a63-8f01-a0b856a4614c-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrt9\" (UniqueName: \"kubernetes.io/projected/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-api-access-2qrt9\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776e97ad-71dd-4a63-8f01-a0b856a4614c-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.225990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:09:28.226100    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:09:28.226170    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls podName:776e97ad-71dd-4a63-8f01-a0b856a4614c nodeName:}" failed. No retries permitted until 2026-09-17 18:09:28.726150831 +0000 UTC m=+2113.173981263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls") pod "isvc-triton-predictor-5d7db58676-xm78g" (UID: "776e97ad-71dd-4a63-8f01-a0b856a4614c") : secret "isvc-triton-predictor-serving-cert" not found
Sep 17 18:09:28.226452 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.226410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.238372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.238290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrt9\" (UniqueName: \"kubernetes.io/projected/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-api-access-2qrt9\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.729900 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.729863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.732285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.732263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xm78g\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:28.901634 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:28.901597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:09:29.180093 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:29.180042    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"]
Sep 17 18:09:29.183141 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:09:29.183116    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776e97ad_71dd_4a63_8f01_a0b856a4614c.slice/crio-3ba411a9e10457d79a1ee3c45dee5197bf649ed5861de221ebaddaadd840aeca WatchSource:0}: Error finding container 3ba411a9e10457d79a1ee3c45dee5197bf649ed5861de221ebaddaadd840aeca: Status 404 returned error can't find the container with id 3ba411a9e10457d79a1ee3c45dee5197bf649ed5861de221ebaddaadd840aeca
Sep 17 18:09:29.518041 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:29.517940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerStarted","Data":"8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4"}
Sep 17 18:09:29.518218 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:29.518048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerStarted","Data":"3ba411a9e10457d79a1ee3c45dee5197bf649ed5861de221ebaddaadd840aeca"}
Sep 17 18:09:33.538134 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:33.538051    2562 generic.go:356] "Generic (PLEG): container finished" podID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerID="8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4" exitCode=0
Sep 17 18:09:33.538134 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:09:33.538122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerDied","Data":"8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4"}
Sep 17 18:10:49.208030 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:49.207812    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"]
Sep 17 18:10:49.231010 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:49.208096    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" containerID="cri-o://d47343c265b815a3995e68bce1eeefadf69d96af85e682bb0143447e7fd32c81" gracePeriod=30
Sep 17 18:10:49.231010 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:49.208366    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kube-rbac-proxy" containerID="cri-o://cf4e3962c2ca44891651059fcc468eafc11cc3e6f629a08301d21a86793f358b" gracePeriod=30
Sep 17 18:10:53.025688 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:53.025653    2562 generic.go:356] "Generic (PLEG): container finished" podID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerID="cf4e3962c2ca44891651059fcc468eafc11cc3e6f629a08301d21a86793f358b" exitCode=2
Sep 17 18:10:53.026067 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:53.025728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerDied","Data":"cf4e3962c2ca44891651059fcc468eafc11cc3e6f629a08301d21a86793f358b"}
Sep 17 18:10:53.061805 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:53.061776    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:10:53.068290 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:53.068269    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:10:54.515489 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.515233    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"]
Sep 17 18:10:54.538491 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.538254    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"]
Sep 17 18:10:54.538491 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.538425    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.541848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.541646    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:10:54.541848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.541808    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:10:54.666422 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.666310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.666629 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.666422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060bbe93-235e-4800-8898-444465bd3a04-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.666629 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.666484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4b2g\" (UniqueName: \"kubernetes.io/projected/060bbe93-235e-4800-8898-444465bd3a04-kube-api-access-f4b2g\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.666629 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.666512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.666629 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.666530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.767297 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.767259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.767594 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.767380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.767594 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.767414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060bbe93-235e-4800-8898-444465bd3a04-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.767594 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.767445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4b2g\" (UniqueName: \"kubernetes.io/projected/060bbe93-235e-4800-8898-444465bd3a04-kube-api-access-f4b2g\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.767594 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.767464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.768368 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:10:54.767985    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:10:54.768368 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:10:54.768052    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls podName:060bbe93-235e-4800-8898-444465bd3a04 nodeName:}" failed. No retries permitted until 2026-09-17 18:10:55.268033091 +0000 UTC m=+2199.715863534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls") pod "isvc-canary-stop-predictor-5cdcf6545-6jrrz" (UID: "060bbe93-235e-4800-8898-444465bd3a04") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:10:54.768368 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.768252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.768633 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.768454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.768633 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.768515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060bbe93-235e-4800-8898-444465bd3a04-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:54.779543 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:54.779515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4b2g\" (UniqueName: \"kubernetes.io/projected/060bbe93-235e-4800-8898-444465bd3a04-kube-api-access-f4b2g\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:55.273900 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:55.273852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:55.276813 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:55.276782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-6jrrz\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:55.454286 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:55.454246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:10:58.061854 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:58.061821    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:10:59.060551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.060519    2562 generic.go:356] "Generic (PLEG): container finished" podID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerID="d47343c265b815a3995e68bce1eeefadf69d96af85e682bb0143447e7fd32c81" exitCode=0
Sep 17 18:10:59.060721 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.060595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerDied","Data":"d47343c265b815a3995e68bce1eeefadf69d96af85e682bb0143447e7fd32c81"}
Sep 17 18:10:59.228403 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.228379    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:10:59.312514 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.312483    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjqpj\" (UniqueName: \"kubernetes.io/projected/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-api-access-gjqpj\") pod \"271b9ec7-60b8-4512-b99e-d44068cc9262\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") "
Sep 17 18:10:59.312623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.312534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-rbac-proxy-sar-config\") pod \"271b9ec7-60b8-4512-b99e-d44068cc9262\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") "
Sep 17 18:10:59.312623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.312553    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-cabundle-cert\") pod \"271b9ec7-60b8-4512-b99e-d44068cc9262\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") "
Sep 17 18:10:59.312623 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.312592    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271b9ec7-60b8-4512-b99e-d44068cc9262-kserve-provision-location\") pod \"271b9ec7-60b8-4512-b99e-d44068cc9262\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") "
Sep 17 18:10:59.312784 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.312668    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls\") pod \"271b9ec7-60b8-4512-b99e-d44068cc9262\" (UID: \"271b9ec7-60b8-4512-b99e-d44068cc9262\") "
Sep 17 18:10:59.313028 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.312982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "271b9ec7-60b8-4512-b99e-d44068cc9262" (UID: "271b9ec7-60b8-4512-b99e-d44068cc9262"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:59.313141 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.313018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "271b9ec7-60b8-4512-b99e-d44068cc9262" (UID: "271b9ec7-60b8-4512-b99e-d44068cc9262"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:59.313141 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.313044    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271b9ec7-60b8-4512-b99e-d44068cc9262-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "271b9ec7-60b8-4512-b99e-d44068cc9262" (UID: "271b9ec7-60b8-4512-b99e-d44068cc9262"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:59.314658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.314635    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-api-access-gjqpj" (OuterVolumeSpecName: "kube-api-access-gjqpj") pod "271b9ec7-60b8-4512-b99e-d44068cc9262" (UID: "271b9ec7-60b8-4512-b99e-d44068cc9262"). InnerVolumeSpecName "kube-api-access-gjqpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:59.314911 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.314894    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "271b9ec7-60b8-4512-b99e-d44068cc9262" (UID: "271b9ec7-60b8-4512-b99e-d44068cc9262"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:59.413650 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.413619    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/271b9ec7-60b8-4512-b99e-d44068cc9262-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:10:59.413650 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.413648    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqpj\" (UniqueName: \"kubernetes.io/projected/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-api-access-gjqpj\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:10:59.413846 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.413661    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:10:59.413846 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.413672    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/271b9ec7-60b8-4512-b99e-d44068cc9262-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:10:59.413846 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.413681    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/271b9ec7-60b8-4512-b99e-d44068cc9262-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:10:59.584738 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:10:59.584617    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"]
Sep 17 18:11:00.068242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:00.068211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592" event={"ID":"271b9ec7-60b8-4512-b99e-d44068cc9262","Type":"ContainerDied","Data":"28e0afffe6525da2111d44448522197fd89e467e286b8373fa80530202d7b567"}
Sep 17 18:11:00.068242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:00.068237    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"
Sep 17 18:11:00.068470 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:00.068283    2562 scope.go:117] "RemoveContainer" containerID="cf4e3962c2ca44891651059fcc468eafc11cc3e6f629a08301d21a86793f358b"
Sep 17 18:11:00.089714 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:00.089688    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"]
Sep 17 18:11:00.094546 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:00.094521    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-wc592"]
Sep 17 18:11:01.728129 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:01.728101    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" path="/var/lib/kubelet/pods/271b9ec7-60b8-4512-b99e-d44068cc9262/volumes"
Sep 17 18:11:27.443444 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:11:27.443404    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060bbe93_235e_4800_8898_444465bd3a04.slice/crio-d761fe9344ff2748435e6b44c6b01417670daa8cbb2b5488b898ceaac9aac7ec WatchSource:0}: Error finding container d761fe9344ff2748435e6b44c6b01417670daa8cbb2b5488b898ceaac9aac7ec: Status 404 returned error can't find the container with id d761fe9344ff2748435e6b44c6b01417670daa8cbb2b5488b898ceaac9aac7ec
Sep 17 18:11:27.709869 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:27.709828    2562 scope.go:117] "RemoveContainer" containerID="d47343c265b815a3995e68bce1eeefadf69d96af85e682bb0143447e7fd32c81"
Sep 17 18:11:27.970392 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:27.970366    2562 scope.go:117] "RemoveContainer" containerID="789fd3da99998d2b9f81f3a44ae4ddb272a102281a8e43564d38118eca8d0d43"
Sep 17 18:11:28.216377 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:28.216318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerStarted","Data":"b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81"}
Sep 17 18:11:28.216597 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:28.216405    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerStarted","Data":"d761fe9344ff2748435e6b44c6b01417670daa8cbb2b5488b898ceaac9aac7ec"}
Sep 17 18:11:29.238229 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:29.238091    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerStarted","Data":"680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a"}
Sep 17 18:11:29.238229 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:29.238196    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:11:29.238777 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:29.238604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:11:29.238777 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:29.238647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerStarted","Data":"0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88"}
Sep 17 18:11:29.239645 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:29.239620    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:11:29.261871 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:29.261799    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podStartSLOduration=2.853404867 podStartE2EDuration="2m2.261782564s" totalImagesPullingTime="1m55.405378238s" totalInitContainerRuntime="4.002999459s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:29.260898647 +0000 UTC m=+2233.708729137" watchObservedRunningTime="2026-09-17 18:11:29.261782564 +0000 UTC m=+2233.709613019"
Sep 17 18:11:30.243559 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:30.243516    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:11:31.248005 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:31.247958    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:11:33.257596 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:33.257563    2562 generic.go:356] "Generic (PLEG): container finished" podID="060bbe93-235e-4800-8898-444465bd3a04" containerID="b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81" exitCode=0
Sep 17 18:11:33.257975 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:33.257617    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerDied","Data":"b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81"}
Sep 17 18:11:34.265430 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:34.265396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerStarted","Data":"3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9"}
Sep 17 18:11:34.265965 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:34.265459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerStarted","Data":"9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532"}
Sep 17 18:11:34.265965 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:34.265751    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:11:34.265965 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:34.265908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:11:34.267247 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:34.267222    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:11:34.290705 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:34.290659    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podStartSLOduration=36.116162549 podStartE2EDuration="40.290646624s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.174484075s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:34.289550595 +0000 UTC m=+2238.737381067" watchObservedRunningTime="2026-09-17 18:11:34.290646624 +0000 UTC m=+2238.738477138"
Sep 17 18:11:35.270823 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:35.270789    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:11:36.253986 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:36.253959    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:11:36.254760 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:36.254741    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:11:40.175426 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.175320    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"]
Sep 17 18:11:40.175817 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.175574    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container" containerID="cri-o://0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88" gracePeriod=30
Sep 17 18:11:40.175817 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.175689    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kube-rbac-proxy" containerID="cri-o://680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a" gracePeriod=30
Sep 17 18:11:40.275446 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.275417    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:11:40.275943 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.275911    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:11:40.298005 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.297974    2562 generic.go:356] "Generic (PLEG): container finished" podID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerID="680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a" exitCode=2
Sep 17 18:11:40.298005 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:40.297988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerDied","Data":"680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a"}
Sep 17 18:11:41.248557 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:41.248520    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:11:42.746624 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:11:42.746591    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776e97ad_71dd_4a63_8f01_a0b856a4614c.slice/crio-0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:11:42.953564 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.953538    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:11:42.974399 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974282    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776e97ad-71dd-4a63-8f01-a0b856a4614c-kserve-provision-location\") pod \"776e97ad-71dd-4a63-8f01-a0b856a4614c\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") "
Sep 17 18:11:42.974399 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-rbac-proxy-sar-config\") pod \"776e97ad-71dd-4a63-8f01-a0b856a4614c\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") "
Sep 17 18:11:42.974608 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974412    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qrt9\" (UniqueName: \"kubernetes.io/projected/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-api-access-2qrt9\") pod \"776e97ad-71dd-4a63-8f01-a0b856a4614c\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") "
Sep 17 18:11:42.974608 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974450    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls\") pod \"776e97ad-71dd-4a63-8f01-a0b856a4614c\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") "
Sep 17 18:11:42.974608 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974539    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-cabundle-cert\") pod \"776e97ad-71dd-4a63-8f01-a0b856a4614c\" (UID: \"776e97ad-71dd-4a63-8f01-a0b856a4614c\") "
Sep 17 18:11:42.974737 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974711    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "776e97ad-71dd-4a63-8f01-a0b856a4614c" (UID: "776e97ad-71dd-4a63-8f01-a0b856a4614c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:42.974737 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974715    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776e97ad-71dd-4a63-8f01-a0b856a4614c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "776e97ad-71dd-4a63-8f01-a0b856a4614c" (UID: "776e97ad-71dd-4a63-8f01-a0b856a4614c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:42.974815 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974784    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/776e97ad-71dd-4a63-8f01-a0b856a4614c-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:11:42.974815 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.974800    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:11:42.975084 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.975063    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "776e97ad-71dd-4a63-8f01-a0b856a4614c" (UID: "776e97ad-71dd-4a63-8f01-a0b856a4614c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:42.976768 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.976740    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-api-access-2qrt9" (OuterVolumeSpecName: "kube-api-access-2qrt9") pod "776e97ad-71dd-4a63-8f01-a0b856a4614c" (UID: "776e97ad-71dd-4a63-8f01-a0b856a4614c"). InnerVolumeSpecName "kube-api-access-2qrt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:42.976918 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:42.976897    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "776e97ad-71dd-4a63-8f01-a0b856a4614c" (UID: "776e97ad-71dd-4a63-8f01-a0b856a4614c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:43.076127 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.076095    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qrt9\" (UniqueName: \"kubernetes.io/projected/776e97ad-71dd-4a63-8f01-a0b856a4614c-kube-api-access-2qrt9\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:11:43.076127 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.076121    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/776e97ad-71dd-4a63-8f01-a0b856a4614c-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:11:43.076127 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.076131    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/776e97ad-71dd-4a63-8f01-a0b856a4614c-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:11:43.313085 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.313051    2562 generic.go:356] "Generic (PLEG): container finished" podID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerID="0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88" exitCode=0
Sep 17 18:11:43.313282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.313116    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"
Sep 17 18:11:43.313282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.313132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerDied","Data":"0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88"}
Sep 17 18:11:43.313282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.313219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g" event={"ID":"776e97ad-71dd-4a63-8f01-a0b856a4614c","Type":"ContainerDied","Data":"3ba411a9e10457d79a1ee3c45dee5197bf649ed5861de221ebaddaadd840aeca"}
Sep 17 18:11:43.313282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.313249    2562 scope.go:117] "RemoveContainer" containerID="680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a"
Sep 17 18:11:43.322069 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.322053    2562 scope.go:117] "RemoveContainer" containerID="0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88"
Sep 17 18:11:43.330162 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.330138    2562 scope.go:117] "RemoveContainer" containerID="8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4"
Sep 17 18:11:43.335805 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.335784    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"]
Sep 17 18:11:43.338337 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.338311    2562 scope.go:117] "RemoveContainer" containerID="680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a"
Sep 17 18:11:43.338629 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:11:43.338611    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a\": container with ID starting with 680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a not found: ID does not exist" containerID="680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a"
Sep 17 18:11:43.338675 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.338637    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a"} err="failed to get container status \"680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a\": rpc error: code = NotFound desc = could not find container \"680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a\": container with ID starting with 680be610dbd3c377f4c8577cff30aab05b2e4f876a0b80e0ab62be6cf4bf270a not found: ID does not exist"
Sep 17 18:11:43.338675 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.338655    2562 scope.go:117] "RemoveContainer" containerID="0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88"
Sep 17 18:11:43.338894 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:11:43.338876    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88\": container with ID starting with 0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88 not found: ID does not exist" containerID="0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88"
Sep 17 18:11:43.338936 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.338900    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88"} err="failed to get container status \"0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88\": rpc error: code = NotFound desc = could not find container \"0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88\": container with ID starting with 0073ecfac27fba47b74c79cea4745a501cdc27fd1ddd8d9a45a4123390641c88 not found: ID does not exist"
Sep 17 18:11:43.338936 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.338920    2562 scope.go:117] "RemoveContainer" containerID="8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4"
Sep 17 18:11:43.339151 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:11:43.339136    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4\": container with ID starting with 8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4 not found: ID does not exist" containerID="8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4"
Sep 17 18:11:43.339192 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.339155    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4"} err="failed to get container status \"8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4\": rpc error: code = NotFound desc = could not find container \"8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4\": container with ID starting with 8cecd08be9a6025ca9dd5ec11e29e1e381745563731e907b80fa0b0cf48f35c4 not found: ID does not exist"
Sep 17 18:11:43.342368 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.342345    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xm78g"]
Sep 17 18:11:43.727927 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:43.727852    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" path="/var/lib/kubelet/pods/776e97ad-71dd-4a63-8f01-a0b856a4614c/volumes"
Sep 17 18:11:45.351195 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351163    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"]
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351500    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kube-rbac-proxy"
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351512    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kube-rbac-proxy"
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351521    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kube-rbac-proxy"
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351527    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kube-rbac-proxy"
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351547    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container"
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351557    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container"
Sep 17 18:11:45.351569 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351570    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="storage-initializer"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351575    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="storage-initializer"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351592    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="storage-initializer"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351597    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="storage-initializer"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351606    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351611    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351658    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kube-rbac-proxy"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351666    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kube-rbac-proxy"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351677    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="776e97ad-71dd-4a63-8f01-a0b856a4614c" containerName="kserve-container"
Sep 17 18:11:45.351848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.351689    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b9ec7-60b8-4512-b99e-d44068cc9262" containerName="kserve-container"
Sep 17 18:11:45.364150 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.364126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.367266 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.367164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"]
Sep 17 18:11:45.372420 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.367524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 18:11:45.372420 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.367656    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 18:11:45.392885 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.392861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3768a304-a4cb-465b-b488-0883b538ae04-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.392991 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.392897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3768a304-a4cb-465b-b488-0883b538ae04-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.392991 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.392966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.392991 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.392990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krhkr\" (UniqueName: \"kubernetes.io/projected/3768a304-a4cb-465b-b488-0883b538ae04-kube-api-access-krhkr\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.393124 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.393038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494115 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494229 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krhkr\" (UniqueName: \"kubernetes.io/projected/3768a304-a4cb-465b-b488-0883b538ae04-kube-api-access-krhkr\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494229 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494404 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3768a304-a4cb-465b-b488-0883b538ae04-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494468 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3768a304-a4cb-465b-b488-0883b538ae04-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494814 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3768a304-a4cb-465b-b488-0883b538ae04-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.494938 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.495004 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.494924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.496864 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.496847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3768a304-a4cb-465b-b488-0883b538ae04-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.507787 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.507764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krhkr\" (UniqueName: \"kubernetes.io/projected/3768a304-a4cb-465b-b488-0883b538ae04-kube-api-access-krhkr\") pod \"isvc-xgboost-predictor-7566d65bf7-k5txw\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.681471 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.681395    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:11:45.982936 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:45.982913    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"]
Sep 17 18:11:45.984454 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:11:45.984427    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3768a304_a4cb_465b_b488_0883b538ae04.slice/crio-abe138b81923b75e2cfc557e1e6bf125f1dc63a4777eb86856b27efeed8dbd1f WatchSource:0}: Error finding container abe138b81923b75e2cfc557e1e6bf125f1dc63a4777eb86856b27efeed8dbd1f: Status 404 returned error can't find the container with id abe138b81923b75e2cfc557e1e6bf125f1dc63a4777eb86856b27efeed8dbd1f
Sep 17 18:11:46.330802 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:46.330768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerStarted","Data":"f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071"}
Sep 17 18:11:46.330948 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:46.330828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerStarted","Data":"abe138b81923b75e2cfc557e1e6bf125f1dc63a4777eb86856b27efeed8dbd1f"}
Sep 17 18:11:50.276769 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:50.276683    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:11:50.349846 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:50.349813    2562 generic.go:356] "Generic (PLEG): container finished" podID="3768a304-a4cb-465b-b488-0883b538ae04" containerID="f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071" exitCode=0
Sep 17 18:11:50.350028 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:11:50.349887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerDied","Data":"f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071"}
Sep 17 18:12:00.277068 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:00.277018    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:12:10.276789 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:10.276748    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:12:11.469705 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:11.469673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerStarted","Data":"44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab"}
Sep 17 18:12:11.470087 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:11.469731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerStarted","Data":"f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31"}
Sep 17 18:12:11.470087 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:11.469917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:12:11.494580 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:11.494536    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podStartSLOduration=1.9268608729999999 podStartE2EDuration="26.494521987s" totalImagesPullingTime="20.647388655s" totalInitContainerRuntime="3.920272459s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:11.492035884 +0000 UTC m=+2275.939866341" watchObservedRunningTime="2026-09-17 18:12:11.494521987 +0000 UTC m=+2275.942352442"
Sep 17 18:12:12.474675 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:12.474630    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:12:12.475848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:12.475819    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:12:13.479389 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:13.479347    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:12:18.483729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:18.483697    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:12:18.484158 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:18.484133    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:12:20.276065 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:20.276027    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:12:28.484992 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:28.484955    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:12:30.276367 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:30.276310    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:12:38.484915 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:38.484877    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:12:40.276525 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:40.276479    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:12:44.722351 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:44.722287    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:12:48.484879 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:48.484838    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:12:54.722552 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:54.722523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:12:58.484415 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:12:58.484378    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:13:04.501576 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:04.501493    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"]
Sep 17 18:13:04.502101 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:04.501781    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" containerID="cri-o://9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532" gracePeriod=30
Sep 17 18:13:04.502101 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:04.501807    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kube-rbac-proxy" containerID="cri-o://3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9" gracePeriod=30
Sep 17 18:13:04.721105 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:04.721071    2562 generic.go:356] "Generic (PLEG): container finished" podID="060bbe93-235e-4800-8898-444465bd3a04" containerID="3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9" exitCode=2
Sep 17 18:13:04.721260 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:04.721154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerDied","Data":"3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9"}
Sep 17 18:13:04.722285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:04.722259    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:13:05.270941 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:05.270900    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 18:13:08.484460 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:08.484422    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:13:09.344718 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.344690    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:13:09.415202 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls\") pod \"060bbe93-235e-4800-8898-444465bd3a04\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") "
Sep 17 18:13:09.415202 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415165    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060bbe93-235e-4800-8898-444465bd3a04-kserve-provision-location\") pod \"060bbe93-235e-4800-8898-444465bd3a04\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") "
Sep 17 18:13:09.415475 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4b2g\" (UniqueName: \"kubernetes.io/projected/060bbe93-235e-4800-8898-444465bd3a04-kube-api-access-f4b2g\") pod \"060bbe93-235e-4800-8898-444465bd3a04\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") "
Sep 17 18:13:09.415475 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-cabundle-cert\") pod \"060bbe93-235e-4800-8898-444465bd3a04\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") "
Sep 17 18:13:09.415475 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415277    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-kube-rbac-proxy-sar-config\") pod \"060bbe93-235e-4800-8898-444465bd3a04\" (UID: \"060bbe93-235e-4800-8898-444465bd3a04\") "
Sep 17 18:13:09.415614 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415542    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060bbe93-235e-4800-8898-444465bd3a04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "060bbe93-235e-4800-8898-444465bd3a04" (UID: "060bbe93-235e-4800-8898-444465bd3a04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:09.415700 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "060bbe93-235e-4800-8898-444465bd3a04" (UID: "060bbe93-235e-4800-8898-444465bd3a04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:09.415739 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.415683    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "060bbe93-235e-4800-8898-444465bd3a04" (UID: "060bbe93-235e-4800-8898-444465bd3a04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:09.417320 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.417301    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "060bbe93-235e-4800-8898-444465bd3a04" (UID: "060bbe93-235e-4800-8898-444465bd3a04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:09.417439 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.417416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060bbe93-235e-4800-8898-444465bd3a04-kube-api-access-f4b2g" (OuterVolumeSpecName: "kube-api-access-f4b2g") pod "060bbe93-235e-4800-8898-444465bd3a04" (UID: "060bbe93-235e-4800-8898-444465bd3a04"). InnerVolumeSpecName "kube-api-access-f4b2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:09.516889 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.516856    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060bbe93-235e-4800-8898-444465bd3a04-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.516889 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.516884    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060bbe93-235e-4800-8898-444465bd3a04-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.516889 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.516895    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4b2g\" (UniqueName: \"kubernetes.io/projected/060bbe93-235e-4800-8898-444465bd3a04-kube-api-access-f4b2g\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.516889 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.516904    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.517379 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.516913    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060bbe93-235e-4800-8898-444465bd3a04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.746118 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.746090    2562 generic.go:356] "Generic (PLEG): container finished" podID="060bbe93-235e-4800-8898-444465bd3a04" containerID="9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532" exitCode=0
Sep 17 18:13:09.746314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.746157    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"
Sep 17 18:13:09.746314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.746165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerDied","Data":"9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532"}
Sep 17 18:13:09.746314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.746222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz" event={"ID":"060bbe93-235e-4800-8898-444465bd3a04","Type":"ContainerDied","Data":"d761fe9344ff2748435e6b44c6b01417670daa8cbb2b5488b898ceaac9aac7ec"}
Sep 17 18:13:09.746314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.746248    2562 scope.go:117] "RemoveContainer" containerID="3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9"
Sep 17 18:13:09.754495 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.754472    2562 scope.go:117] "RemoveContainer" containerID="9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532"
Sep 17 18:13:09.762126 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.762103    2562 scope.go:117] "RemoveContainer" containerID="b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81"
Sep 17 18:13:09.765194 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.765171    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"]
Sep 17 18:13:09.770168 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.770151    2562 scope.go:117] "RemoveContainer" containerID="3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9"
Sep 17 18:13:09.770475 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:13:09.770457    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9\": container with ID starting with 3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9 not found: ID does not exist" containerID="3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9"
Sep 17 18:13:09.770549 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.770482    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9"} err="failed to get container status \"3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9\": rpc error: code = NotFound desc = could not find container \"3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9\": container with ID starting with 3278a5756c9bf161d60f2af7ef39466c568f8a9c1cb378a799304f95ce8f70c9 not found: ID does not exist"
Sep 17 18:13:09.770549 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.770500    2562 scope.go:117] "RemoveContainer" containerID="9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532"
Sep 17 18:13:09.770732 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:13:09.770716    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532\": container with ID starting with 9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532 not found: ID does not exist" containerID="9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532"
Sep 17 18:13:09.770795 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.770736    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532"} err="failed to get container status \"9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532\": rpc error: code = NotFound desc = could not find container \"9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532\": container with ID starting with 9f33122675a809d0a10560af857793ee978e8e919cfe21f82c4eb5c8436dd532 not found: ID does not exist"
Sep 17 18:13:09.770795 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.770750    2562 scope.go:117] "RemoveContainer" containerID="b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81"
Sep 17 18:13:09.770995 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:13:09.770974    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81\": container with ID starting with b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81 not found: ID does not exist" containerID="b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81"
Sep 17 18:13:09.771051 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.771002    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81"} err="failed to get container status \"b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81\": rpc error: code = NotFound desc = could not find container \"b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81\": container with ID starting with b5b989566c9f565d0172a24ce6a3cbcd4814405d816fbb3fd3b5d3c312017e81 not found: ID does not exist"
Sep 17 18:13:09.771108 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:09.771069    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-6jrrz"]
Sep 17 18:13:11.727618 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:11.727587    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060bbe93-235e-4800-8898-444465bd3a04" path="/var/lib/kubelet/pods/060bbe93-235e-4800-8898-444465bd3a04/volumes"
Sep 17 18:13:18.485486 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:18.485457    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:13:25.854739 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:25.854709    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"]
Sep 17 18:13:25.855143 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:25.854940    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" containerID="cri-o://f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31" gracePeriod=30
Sep 17 18:13:25.855143 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:25.854982    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kube-rbac-proxy" containerID="cri-o://44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab" gracePeriod=30
Sep 17 18:13:26.824016 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:26.823980    2562 generic.go:356] "Generic (PLEG): container finished" podID="3768a304-a4cb-465b-b488-0883b538ae04" containerID="44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab" exitCode=2
Sep 17 18:13:26.824200 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:26.824056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerDied","Data":"44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab"}
Sep 17 18:13:28.480323 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:28.480279    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 17 18:13:28.484654 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:28.484626    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 18:13:29.700027 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.700005    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:13:29.838780 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.838750    2562 generic.go:356] "Generic (PLEG): container finished" podID="3768a304-a4cb-465b-b488-0883b538ae04" containerID="f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31" exitCode=0
Sep 17 18:13:29.838908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.838791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerDied","Data":"f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31"}
Sep 17 18:13:29.838908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.838804    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"
Sep 17 18:13:29.838908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.838837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw" event={"ID":"3768a304-a4cb-465b-b488-0883b538ae04","Type":"ContainerDied","Data":"abe138b81923b75e2cfc557e1e6bf125f1dc63a4777eb86856b27efeed8dbd1f"}
Sep 17 18:13:29.838908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.838862    2562 scope.go:117] "RemoveContainer" containerID="44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab"
Sep 17 18:13:29.846792 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.846777    2562 scope.go:117] "RemoveContainer" containerID="f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31"
Sep 17 18:13:29.854083 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.854071    2562 scope.go:117] "RemoveContainer" containerID="f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071"
Sep 17 18:13:29.861046 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.861031    2562 scope.go:117] "RemoveContainer" containerID="44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab"
Sep 17 18:13:29.861284 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:13:29.861268    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab\": container with ID starting with 44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab not found: ID does not exist" containerID="44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab"
Sep 17 18:13:29.861354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.861292    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab"} err="failed to get container status \"44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab\": rpc error: code = NotFound desc = could not find container \"44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab\": container with ID starting with 44ef1f06fc101e7523fe0e5579cb2f375d444d27d4c8446fb69460ce601d0aab not found: ID does not exist"
Sep 17 18:13:29.861354 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.861308    2562 scope.go:117] "RemoveContainer" containerID="f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31"
Sep 17 18:13:29.861542 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:13:29.861526    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31\": container with ID starting with f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31 not found: ID does not exist" containerID="f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31"
Sep 17 18:13:29.861598 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.861548    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31"} err="failed to get container status \"f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31\": rpc error: code = NotFound desc = could not find container \"f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31\": container with ID starting with f84d1dae372fc3c0c63c6735d3d899a9cebd7dd48cb9a7f92f596dfe57c09e31 not found: ID does not exist"
Sep 17 18:13:29.861598 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.861561    2562 scope.go:117] "RemoveContainer" containerID="f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071"
Sep 17 18:13:29.861790 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:13:29.861772    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071\": container with ID starting with f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071 not found: ID does not exist" containerID="f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071"
Sep 17 18:13:29.861841 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.861798    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071"} err="failed to get container status \"f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071\": rpc error: code = NotFound desc = could not find container \"f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071\": container with ID starting with f46f2fbd31b7b8c642ddb3e11759c97320fff9238450d0c88d2ff405fa9b4071 not found: ID does not exist"
Sep 17 18:13:29.886083 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886058    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-kube-rbac-proxy-sar-config\") pod \"3768a304-a4cb-465b-b488-0883b538ae04\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") "
Sep 17 18:13:29.886177 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886094    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krhkr\" (UniqueName: \"kubernetes.io/projected/3768a304-a4cb-465b-b488-0883b538ae04-kube-api-access-krhkr\") pod \"3768a304-a4cb-465b-b488-0883b538ae04\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") "
Sep 17 18:13:29.886177 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886132    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3768a304-a4cb-465b-b488-0883b538ae04-kserve-provision-location\") pod \"3768a304-a4cb-465b-b488-0883b538ae04\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") "
Sep 17 18:13:29.886280 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-cabundle-cert\") pod \"3768a304-a4cb-465b-b488-0883b538ae04\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") "
Sep 17 18:13:29.886280 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886212    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3768a304-a4cb-465b-b488-0883b538ae04-proxy-tls\") pod \"3768a304-a4cb-465b-b488-0883b538ae04\" (UID: \"3768a304-a4cb-465b-b488-0883b538ae04\") "
Sep 17 18:13:29.886454 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3768a304-a4cb-465b-b488-0883b538ae04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3768a304-a4cb-465b-b488-0883b538ae04" (UID: "3768a304-a4cb-465b-b488-0883b538ae04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:29.886500 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886458    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3768a304-a4cb-465b-b488-0883b538ae04" (UID: "3768a304-a4cb-465b-b488-0883b538ae04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:29.886560 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886546    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:29.886594 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886572    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3768a304-a4cb-465b-b488-0883b538ae04-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:29.886594 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.886543    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3768a304-a4cb-465b-b488-0883b538ae04" (UID: "3768a304-a4cb-465b-b488-0883b538ae04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:29.888058 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.888031    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3768a304-a4cb-465b-b488-0883b538ae04-kube-api-access-krhkr" (OuterVolumeSpecName: "kube-api-access-krhkr") pod "3768a304-a4cb-465b-b488-0883b538ae04" (UID: "3768a304-a4cb-465b-b488-0883b538ae04"). InnerVolumeSpecName "kube-api-access-krhkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:29.888162 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.888139    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3768a304-a4cb-465b-b488-0883b538ae04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3768a304-a4cb-465b-b488-0883b538ae04" (UID: "3768a304-a4cb-465b-b488-0883b538ae04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:29.987651 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.987621    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krhkr\" (UniqueName: \"kubernetes.io/projected/3768a304-a4cb-465b-b488-0883b538ae04-kube-api-access-krhkr\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:29.987651 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.987651    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3768a304-a4cb-465b-b488-0883b538ae04-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:29.987651 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:29.987661    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3768a304-a4cb-465b-b488-0883b538ae04-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:13:30.162928 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:30.162903    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"]
Sep 17 18:13:30.167827 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:30.167807    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k5txw"]
Sep 17 18:13:31.096422 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.096387    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"]
Sep 17 18:13:31.097437 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097417    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container"
Sep 17 18:13:31.097543 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097534    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container"
Sep 17 18:13:31.097640 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097629    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="storage-initializer"
Sep 17 18:13:31.097729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097720    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="storage-initializer"
Sep 17 18:13:31.097820 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097811    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container"
Sep 17 18:13:31.097904 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097896    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container"
Sep 17 18:13:31.098003 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.097994    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="storage-initializer"
Sep 17 18:13:31.098078 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098070    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="storage-initializer"
Sep 17 18:13:31.098153 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098145    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kube-rbac-proxy"
Sep 17 18:13:31.098224 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098216    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kube-rbac-proxy"
Sep 17 18:13:31.098308 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098300    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kube-rbac-proxy"
Sep 17 18:13:31.098400 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098390    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kube-rbac-proxy"
Sep 17 18:13:31.098582 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098560    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kube-rbac-proxy"
Sep 17 18:13:31.098582 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098582    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kserve-container"
Sep 17 18:13:31.098747 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098592    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3768a304-a4cb-465b-b488-0883b538ae04" containerName="kserve-container"
Sep 17 18:13:31.098747 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.098605    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="060bbe93-235e-4800-8898-444465bd3a04" containerName="kube-rbac-proxy"
Sep 17 18:13:31.103572 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.103551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.106887 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.106426    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:13:31.106887 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.106832    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:13:31.107990 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.107080    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:13:31.107990 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.107838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:13:31.108167 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.108091    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:13:31.108440 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.108421    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:13:31.108666 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.108643    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:13:31.117839 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.117773    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"]
Sep 17 18:13:31.295789 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.295754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.295789 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.295794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntt4w\" (UniqueName: \"kubernetes.io/projected/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-api-access-ntt4w\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.296010 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.295818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.296010 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.295887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92abf279-abd0-4e96-8d19-53b78545f8e0-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.296010 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.295919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92abf279-abd0-4e96-8d19-53b78545f8e0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.396947 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.396863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92abf279-abd0-4e96-8d19-53b78545f8e0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.396947 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.396917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.397181 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.396956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntt4w\" (UniqueName: \"kubernetes.io/projected/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-api-access-ntt4w\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.397181 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.396980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.397181 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.397028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92abf279-abd0-4e96-8d19-53b78545f8e0-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.397433 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.397412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92abf279-abd0-4e96-8d19-53b78545f8e0-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.397669 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.397648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.397728 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.397647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.399363 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.399320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92abf279-abd0-4e96-8d19-53b78545f8e0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.407597 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.407569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntt4w\" (UniqueName: \"kubernetes.io/projected/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-api-access-ntt4w\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.419254 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.419231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:31.702308 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.702104    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"]
Sep 17 18:13:31.704879 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:13:31.704847    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92abf279_abd0_4e96_8d19_53b78545f8e0.slice/crio-ff3ccfafa416cd4d0203cde732ae96dd96981c73f7d05e05e43d0b910a78dcaa WatchSource:0}: Error finding container ff3ccfafa416cd4d0203cde732ae96dd96981c73f7d05e05e43d0b910a78dcaa: Status 404 returned error can't find the container with id ff3ccfafa416cd4d0203cde732ae96dd96981c73f7d05e05e43d0b910a78dcaa
Sep 17 18:13:31.728876 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.728852    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3768a304-a4cb-465b-b488-0883b538ae04" path="/var/lib/kubelet/pods/3768a304-a4cb-465b-b488-0883b538ae04/volumes"
Sep 17 18:13:31.850794 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.850762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerStarted","Data":"cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053"}
Sep 17 18:13:31.850931 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:31.850816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerStarted","Data":"ff3ccfafa416cd4d0203cde732ae96dd96981c73f7d05e05e43d0b910a78dcaa"}
Sep 17 18:13:35.869610 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:35.869567    2562 generic.go:356] "Generic (PLEG): container finished" podID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerID="cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053" exitCode=0
Sep 17 18:13:35.869936 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:35.869637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerDied","Data":"cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053"}
Sep 17 18:13:36.876023 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:36.875992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerStarted","Data":"87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2"}
Sep 17 18:13:36.876546 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:36.876062    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerStarted","Data":"cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7"}
Sep 17 18:13:36.876546 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:36.876281    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:36.902125 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:36.902071    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" podStartSLOduration=2.033231834 podStartE2EDuration="5.902058634s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8688268s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:36.90000762 +0000 UTC m=+2361.347838078" watchObservedRunningTime="2026-09-17 18:13:36.902058634 +0000 UTC m=+2361.349889089"
Sep 17 18:13:37.880285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:37.880253    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:13:43.891445 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:13:43.891418    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:14:13.895921 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:13.895891    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:14:17.857676 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:17.857648    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:14:17.863321 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:17.863295    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:14:21.107059 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:21.107027    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"]
Sep 17 18:14:21.107545 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:21.107268    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kserve-container" containerID="cri-o://cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7" gracePeriod=30
Sep 17 18:14:21.107545 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:21.107297    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kube-rbac-proxy" containerID="cri-o://87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2" gracePeriod=30
Sep 17 18:14:22.079635 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:22.079601    2562 generic.go:356] "Generic (PLEG): container finished" podID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerID="87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2" exitCode=2
Sep 17 18:14:22.079798 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:22.079670    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerDied","Data":"87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2"}
Sep 17 18:14:23.886498 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:23.886461    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 17 18:14:23.891882 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:23.891858    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:14:26.299762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.299726    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"]
Sep 17 18:14:26.303959 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.303941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.306231 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.306210    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:14:26.306442 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.306424    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:14:26.313455 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.313433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"]
Sep 17 18:14:26.350889 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.350861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.351032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.350896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.351032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.350934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.351032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.350982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv278\" (UniqueName: \"kubernetes.io/projected/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-api-access-tv278\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.351032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.351004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158cfc11-d42e-4540-b50d-1d3f96111ab4-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.451709 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.451670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.451709 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.451717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.451954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.451762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.451954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.451784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv278\" (UniqueName: \"kubernetes.io/projected/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-api-access-tv278\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.451954 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.451807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158cfc11-d42e-4540-b50d-1d3f96111ab4-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.451954 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:14:26.451911    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:14:26.452155 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:14:26.451985    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls podName:158cfc11-d42e-4540-b50d-1d3f96111ab4 nodeName:}" failed. No retries permitted until 2026-09-17 18:14:26.951963526 +0000 UTC m=+2411.399793984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" (UID: "158cfc11-d42e-4540-b50d-1d3f96111ab4") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:14:26.452229 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.452180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158cfc11-d42e-4540-b50d-1d3f96111ab4-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.452432 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.452414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.452536 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.452519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.460424 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.460407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv278\" (UniqueName: \"kubernetes.io/projected/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-api-access-tv278\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.955902 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.955869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:26.958405 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:26.958381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:27.048047 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.048024    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:14:27.056625 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.056601    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92abf279-abd0-4e96-8d19-53b78545f8e0-kserve-provision-location\") pod \"92abf279-abd0-4e96-8d19-53b78545f8e0\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") "
Sep 17 18:14:27.056719 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.056649    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-cabundle-cert\") pod \"92abf279-abd0-4e96-8d19-53b78545f8e0\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") "
Sep 17 18:14:27.056719 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.056674    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-rbac-proxy-sar-config\") pod \"92abf279-abd0-4e96-8d19-53b78545f8e0\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") "
Sep 17 18:14:27.056817 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.056730    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92abf279-abd0-4e96-8d19-53b78545f8e0-proxy-tls\") pod \"92abf279-abd0-4e96-8d19-53b78545f8e0\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") "
Sep 17 18:14:27.056817 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.056758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntt4w\" (UniqueName: \"kubernetes.io/projected/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-api-access-ntt4w\") pod \"92abf279-abd0-4e96-8d19-53b78545f8e0\" (UID: \"92abf279-abd0-4e96-8d19-53b78545f8e0\") "
Sep 17 18:14:27.056964 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.056870    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92abf279-abd0-4e96-8d19-53b78545f8e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92abf279-abd0-4e96-8d19-53b78545f8e0" (UID: "92abf279-abd0-4e96-8d19-53b78545f8e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:27.057035 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.057012    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92abf279-abd0-4e96-8d19-53b78545f8e0" (UID: "92abf279-abd0-4e96-8d19-53b78545f8e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:27.057085 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.057053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92abf279-abd0-4e96-8d19-53b78545f8e0" (UID: "92abf279-abd0-4e96-8d19-53b78545f8e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:27.058736 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.058714    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-api-access-ntt4w" (OuterVolumeSpecName: "kube-api-access-ntt4w") pod "92abf279-abd0-4e96-8d19-53b78545f8e0" (UID: "92abf279-abd0-4e96-8d19-53b78545f8e0"). InnerVolumeSpecName "kube-api-access-ntt4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:27.058810 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.058738    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92abf279-abd0-4e96-8d19-53b78545f8e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92abf279-abd0-4e96-8d19-53b78545f8e0" (UID: "92abf279-abd0-4e96-8d19-53b78545f8e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:27.104375 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.104298    2562 generic.go:356] "Generic (PLEG): container finished" podID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerID="cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7" exitCode=0
Sep 17 18:14:27.104470 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.104386    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"
Sep 17 18:14:27.104470 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.104385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerDied","Data":"cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7"}
Sep 17 18:14:27.104470 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.104446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5" event={"ID":"92abf279-abd0-4e96-8d19-53b78545f8e0","Type":"ContainerDied","Data":"ff3ccfafa416cd4d0203cde732ae96dd96981c73f7d05e05e43d0b910a78dcaa"}
Sep 17 18:14:27.104571 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.104476    2562 scope.go:117] "RemoveContainer" containerID="87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2"
Sep 17 18:14:27.113010 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.112993    2562 scope.go:117] "RemoveContainer" containerID="cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7"
Sep 17 18:14:27.120449 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.120427    2562 scope.go:117] "RemoveContainer" containerID="cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053"
Sep 17 18:14:27.125848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.125821    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"]
Sep 17 18:14:27.128233 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.128214    2562 scope.go:117] "RemoveContainer" containerID="87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2"
Sep 17 18:14:27.128564 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:14:27.128537    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2\": container with ID starting with 87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2 not found: ID does not exist" containerID="87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2"
Sep 17 18:14:27.128647 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.128573    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2"} err="failed to get container status \"87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2\": rpc error: code = NotFound desc = could not find container \"87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2\": container with ID starting with 87f0847d93290db8eef26884857511d9a32ea19e4935a2daeae8fb1ad8f0a1e2 not found: ID does not exist"
Sep 17 18:14:27.128647 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.128595    2562 scope.go:117] "RemoveContainer" containerID="cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7"
Sep 17 18:14:27.128898 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:14:27.128879    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7\": container with ID starting with cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7 not found: ID does not exist" containerID="cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7"
Sep 17 18:14:27.128958 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.128903    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7"} err="failed to get container status \"cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7\": rpc error: code = NotFound desc = could not find container \"cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7\": container with ID starting with cfd8d936e984e842a0f8026bb75b252441358fa4b7750d34e0419f742528c1b7 not found: ID does not exist"
Sep 17 18:14:27.128958 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.128918    2562 scope.go:117] "RemoveContainer" containerID="cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053"
Sep 17 18:14:27.129157 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:14:27.129137    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053\": container with ID starting with cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053 not found: ID does not exist" containerID="cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053"
Sep 17 18:14:27.129199 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.129165    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053"} err="failed to get container status \"cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053\": rpc error: code = NotFound desc = could not find container \"cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053\": container with ID starting with cef6dae78e97aef426336ea7d43ac94cc5ac6e3cee1e1126cb621ab551b65053 not found: ID does not exist"
Sep 17 18:14:27.129434 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.129420    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ftvh5"]
Sep 17 18:14:27.157583 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.157563    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92abf279-abd0-4e96-8d19-53b78545f8e0-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:14:27.157660 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.157585    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:14:27.157660 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.157599    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:14:27.157660 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.157607    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92abf279-abd0-4e96-8d19-53b78545f8e0-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:14:27.157660 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.157617    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntt4w\" (UniqueName: \"kubernetes.io/projected/92abf279-abd0-4e96-8d19-53b78545f8e0-kube-api-access-ntt4w\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:14:27.217001 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.216977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:27.494789 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.492110    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"]
Sep 17 18:14:27.499299 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:14:27.499272    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158cfc11_d42e_4540_b50d_1d3f96111ab4.slice/crio-078f0973d4c7e438f71cc84dc5b4e7d3b325acb893477ce8cc2770e3900cf96b WatchSource:0}: Error finding container 078f0973d4c7e438f71cc84dc5b4e7d3b325acb893477ce8cc2770e3900cf96b: Status 404 returned error can't find the container with id 078f0973d4c7e438f71cc84dc5b4e7d3b325acb893477ce8cc2770e3900cf96b
Sep 17 18:14:27.727920 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:27.727832    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" path="/var/lib/kubelet/pods/92abf279-abd0-4e96-8d19-53b78545f8e0/volumes"
Sep 17 18:14:28.113101 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:28.113067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerStarted","Data":"5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307"}
Sep 17 18:14:28.113303 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:28.113121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerStarted","Data":"078f0973d4c7e438f71cc84dc5b4e7d3b325acb893477ce8cc2770e3900cf96b"}
Sep 17 18:14:32.131706 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:32.131674    2562 generic.go:356] "Generic (PLEG): container finished" podID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerID="5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307" exitCode=0
Sep 17 18:14:32.132059 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:32.131757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerDied","Data":"5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307"}
Sep 17 18:14:33.139998 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:33.139970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerStarted","Data":"4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f"}
Sep 17 18:14:33.140387 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:33.140026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerStarted","Data":"c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c"}
Sep 17 18:14:33.140387 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:33.140197    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:33.161427 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:33.161382    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" podStartSLOduration=3.344754086 podStartE2EDuration="7.161367032s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.816612946s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:33.1600837 +0000 UTC m=+2417.607914153" watchObservedRunningTime="2026-09-17 18:14:33.161367032 +0000 UTC m=+2417.609197485"
Sep 17 18:14:34.144053 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:34.144023    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:14:40.153818 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:14:40.153789    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:15:10.157347 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:10.157300    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:15:16.359113 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:16.359078    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"]
Sep 17 18:15:16.359551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:16.359346    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kserve-container" containerID="cri-o://c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c" gracePeriod=30
Sep 17 18:15:16.359551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:16.359403    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kube-rbac-proxy" containerID="cri-o://4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f" gracePeriod=30
Sep 17 18:15:17.340924 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:17.340889    2562 generic.go:356] "Generic (PLEG): container finished" podID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerID="4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f" exitCode=2
Sep 17 18:15:17.341100 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:17.340965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerDied","Data":"4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f"}
Sep 17 18:15:20.148817 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:20.148775    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 17 18:15:20.154418 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:20.154388    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.52:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:15:22.694771 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.694747    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:15:22.800444 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800366    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv278\" (UniqueName: \"kubernetes.io/projected/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-api-access-tv278\") pod \"158cfc11-d42e-4540-b50d-1d3f96111ab4\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") "
Sep 17 18:15:22.800444 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800423    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls\") pod \"158cfc11-d42e-4540-b50d-1d3f96111ab4\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") "
Sep 17 18:15:22.800686 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800457    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158cfc11-d42e-4540-b50d-1d3f96111ab4-kserve-provision-location\") pod \"158cfc11-d42e-4540-b50d-1d3f96111ab4\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") "
Sep 17 18:15:22.800686 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-cabundle-cert\") pod \"158cfc11-d42e-4540-b50d-1d3f96111ab4\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") "
Sep 17 18:15:22.800686 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800592    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-rbac-proxy-sar-config\") pod \"158cfc11-d42e-4540-b50d-1d3f96111ab4\" (UID: \"158cfc11-d42e-4540-b50d-1d3f96111ab4\") "
Sep 17 18:15:22.800831 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800779    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158cfc11-d42e-4540-b50d-1d3f96111ab4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "158cfc11-d42e-4540-b50d-1d3f96111ab4" (UID: "158cfc11-d42e-4540-b50d-1d3f96111ab4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:22.800951 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "158cfc11-d42e-4540-b50d-1d3f96111ab4" (UID: "158cfc11-d42e-4540-b50d-1d3f96111ab4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:22.801018 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.800992    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "158cfc11-d42e-4540-b50d-1d3f96111ab4" (UID: "158cfc11-d42e-4540-b50d-1d3f96111ab4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:22.802448 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.802422    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-api-access-tv278" (OuterVolumeSpecName: "kube-api-access-tv278") pod "158cfc11-d42e-4540-b50d-1d3f96111ab4" (UID: "158cfc11-d42e-4540-b50d-1d3f96111ab4"). InnerVolumeSpecName "kube-api-access-tv278". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:22.802558 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.802461    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "158cfc11-d42e-4540-b50d-1d3f96111ab4" (UID: "158cfc11-d42e-4540-b50d-1d3f96111ab4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:22.901613 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.901590    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.901613 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.901609    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv278\" (UniqueName: \"kubernetes.io/projected/158cfc11-d42e-4540-b50d-1d3f96111ab4-kube-api-access-tv278\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.901763 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.901620    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158cfc11-d42e-4540-b50d-1d3f96111ab4-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.901763 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.901630    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158cfc11-d42e-4540-b50d-1d3f96111ab4-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.901763 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:22.901639    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/158cfc11-d42e-4540-b50d-1d3f96111ab4-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:15:23.369567 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.369535    2562 generic.go:356] "Generic (PLEG): container finished" podID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerID="c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c" exitCode=0
Sep 17 18:15:23.369731 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.369598    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"
Sep 17 18:15:23.369731 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.369613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerDied","Data":"c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c"}
Sep 17 18:15:23.369731 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.369669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7" event={"ID":"158cfc11-d42e-4540-b50d-1d3f96111ab4","Type":"ContainerDied","Data":"078f0973d4c7e438f71cc84dc5b4e7d3b325acb893477ce8cc2770e3900cf96b"}
Sep 17 18:15:23.369731 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.369695    2562 scope.go:117] "RemoveContainer" containerID="4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f"
Sep 17 18:15:23.378109 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.378092    2562 scope.go:117] "RemoveContainer" containerID="c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c"
Sep 17 18:15:23.385871 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.385854    2562 scope.go:117] "RemoveContainer" containerID="5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307"
Sep 17 18:15:23.391556 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.391537    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"]
Sep 17 18:15:23.393377 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.393359    2562 scope.go:117] "RemoveContainer" containerID="4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f"
Sep 17 18:15:23.393711 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:15:23.393683    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f\": container with ID starting with 4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f not found: ID does not exist" containerID="4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f"
Sep 17 18:15:23.393848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.393727    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f"} err="failed to get container status \"4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f\": rpc error: code = NotFound desc = could not find container \"4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f\": container with ID starting with 4ea943abe3fda344d6a65e080d104413dcf149cddc0fe03da302adfdc779560f not found: ID does not exist"
Sep 17 18:15:23.393848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.393751    2562 scope.go:117] "RemoveContainer" containerID="c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c"
Sep 17 18:15:23.394042 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:15:23.394024    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c\": container with ID starting with c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c not found: ID does not exist" containerID="c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c"
Sep 17 18:15:23.394090 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.394047    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c"} err="failed to get container status \"c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c\": rpc error: code = NotFound desc = could not find container \"c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c\": container with ID starting with c696662e27444cafe637e02ad07dab2bdebb5976f6f9f5ce17a6258328d1a36c not found: ID does not exist"
Sep 17 18:15:23.394090 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.394063    2562 scope.go:117] "RemoveContainer" containerID="5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307"
Sep 17 18:15:23.394303 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:15:23.394289    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307\": container with ID starting with 5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307 not found: ID does not exist" containerID="5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307"
Sep 17 18:15:23.394372 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.394306    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307"} err="failed to get container status \"5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307\": rpc error: code = NotFound desc = could not find container \"5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307\": container with ID starting with 5a4022fac550793c08c19c400bb599946d1fd51851d545ed7382ea5270972307 not found: ID does not exist"
Sep 17 18:15:23.395797 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.395777    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-nmzc7"]
Sep 17 18:15:23.727192 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:15:23.727117    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" path="/var/lib/kubelet/pods/158cfc11-d42e-4540-b50d-1d3f96111ab4/volumes"
Sep 17 18:17:07.346769 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.346729    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"]
Sep 17 18:17:07.347215 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347192    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kube-rbac-proxy"
Sep 17 18:17:07.347215 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347205    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kube-rbac-proxy"
Sep 17 18:17:07.347293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347224    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="storage-initializer"
Sep 17 18:17:07.347293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347234    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="storage-initializer"
Sep 17 18:17:07.347293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347246    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="storage-initializer"
Sep 17 18:17:07.347293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347255    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="storage-initializer"
Sep 17 18:17:07.347293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347280    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kserve-container"
Sep 17 18:17:07.347293 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347288    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kserve-container"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347297    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kube-rbac-proxy"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347305    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kube-rbac-proxy"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347351    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kserve-container"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347360    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kserve-container"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347431    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kserve-container"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347443    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kserve-container"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347453    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="92abf279-abd0-4e96-8d19-53b78545f8e0" containerName="kube-rbac-proxy"
Sep 17 18:17:07.347501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.347461    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="158cfc11-d42e-4540-b50d-1d3f96111ab4" containerName="kube-rbac-proxy"
Sep 17 18:17:07.351242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.351223    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.354525 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.354468    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:17:07.354525 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.354485    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:17:07.354525 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.354485    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:17:07.354747 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.354547    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:17:07.355246 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.355229    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 18:17:07.355246 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.355239    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:17:07.355535 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.355520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:17:07.362364 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.362342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"]
Sep 17 18:17:07.465682 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.465648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.465878 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.465698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24bx\" (UniqueName: \"kubernetes.io/projected/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-api-access-z24bx\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.465878 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.465718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.465878 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.465773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.465878 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.465868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82125b9d-5466-4a1d-a8a0-e665baa0fa48-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.566614 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.566577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.566819 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.566631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24bx\" (UniqueName: \"kubernetes.io/projected/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-api-access-z24bx\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.566819 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.566651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.566819 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.566673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.566819 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.566711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82125b9d-5466-4a1d-a8a0-e665baa0fa48-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.567069 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.567044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.567393 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.567322    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.567483 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.567400    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.569042 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.569025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82125b9d-5466-4a1d-a8a0-e665baa0fa48-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.577632 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.577607    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24bx\" (UniqueName: \"kubernetes.io/projected/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-api-access-z24bx\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.662419 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.662303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:07.948164 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:07.948132    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"]
Sep 17 18:17:07.949223 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:17:07.949189    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82125b9d_5466_4a1d_a8a0_e665baa0fa48.slice/crio-72ce1ae1b0f69b3cc616b95dd58afe52f09fc867a67d294b0d1578ea4f828080 WatchSource:0}: Error finding container 72ce1ae1b0f69b3cc616b95dd58afe52f09fc867a67d294b0d1578ea4f828080: Status 404 returned error can't find the container with id 72ce1ae1b0f69b3cc616b95dd58afe52f09fc867a67d294b0d1578ea4f828080
Sep 17 18:17:08.835192 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:08.835152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerStarted","Data":"5113ca25201ccb168cb6c8e145e3ca3d00db1b8c6d4943cd5e8019cc4c0ad1e3"}
Sep 17 18:17:08.835610 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:08.835239    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerStarted","Data":"72ce1ae1b0f69b3cc616b95dd58afe52f09fc867a67d294b0d1578ea4f828080"}
Sep 17 18:17:11.850607 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:11.850576    2562 generic.go:356] "Generic (PLEG): container finished" podID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerID="5113ca25201ccb168cb6c8e145e3ca3d00db1b8c6d4943cd5e8019cc4c0ad1e3" exitCode=0
Sep 17 18:17:11.851014 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:11.850654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerDied","Data":"5113ca25201ccb168cb6c8e145e3ca3d00db1b8c6d4943cd5e8019cc4c0ad1e3"}
Sep 17 18:17:12.857511 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:12.857469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerStarted","Data":"d07ec787ed483895013c8c93adf45d9001ea0fb2e12de609fba60028ad23ea44"}
Sep 17 18:17:12.857908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:12.857535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerStarted","Data":"f0a94ef2b5ee8fc576d966b219aa7e1fde904897eca94e75ed41eb7f18a1cda4"}
Sep 17 18:17:12.857908 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:12.857733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:12.880453 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:12.880394    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" podStartSLOduration=2.205893926 podStartE2EDuration="5.880376376s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.67448245s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:12.878779518 +0000 UTC m=+2577.326609992" watchObservedRunningTime="2026-09-17 18:17:12.880376376 +0000 UTC m=+2577.328206827"
Sep 17 18:17:13.862222 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:13.862194    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:19.872130 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:19.872101    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:17:20.802982 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.802949    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"]
Sep 17 18:17:20.806382 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.806361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:20.808730 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.808711    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 18:17:20.808891 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.808777    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:17:20.816353 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.816321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"]
Sep 17 18:17:20.986725 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.986688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:20.986725 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.986729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:20.987147 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.986753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:20.987147 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.986886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:20.987147 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:20.986921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrg8v\" (UniqueName: \"kubernetes.io/projected/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-api-access-vrg8v\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.087626 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.087592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.087626 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.087625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg8v\" (UniqueName: \"kubernetes.io/projected/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-api-access-vrg8v\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.087836 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.087672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.087836 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.087704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.087836 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.087731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.088182 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.088161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.088275 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.088249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.088427 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.088407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.090055 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.090034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.096173 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.096148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg8v\" (UniqueName: \"kubernetes.io/projected/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-api-access-vrg8v\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.116918 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.116896    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:21.399120 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.399044    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"]
Sep 17 18:17:21.402449 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:17:21.402425    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0ba285_3160_4ac7_bdff_b28d646dd9bb.slice/crio-a68abe7783d0891d4b10941eeadfa23ca658349e13247c6403d769d710a2b41b WatchSource:0}: Error finding container a68abe7783d0891d4b10941eeadfa23ca658349e13247c6403d769d710a2b41b: Status 404 returned error can't find the container with id a68abe7783d0891d4b10941eeadfa23ca658349e13247c6403d769d710a2b41b
Sep 17 18:17:21.901732 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.901690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerStarted","Data":"f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4"}
Sep 17 18:17:21.901909 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:21.901755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerStarted","Data":"a68abe7783d0891d4b10941eeadfa23ca658349e13247c6403d769d710a2b41b"}
Sep 17 18:17:25.920611 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:25.920572    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerID="f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4" exitCode=0
Sep 17 18:17:25.921073 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:25.920627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerDied","Data":"f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4"}
Sep 17 18:17:26.928199 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:26.928164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerStarted","Data":"374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284"}
Sep 17 18:17:26.928664 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:26.928233    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerStarted","Data":"32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795"}
Sep 17 18:17:26.928664 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:26.928472    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:26.949190 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:26.949142    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" podStartSLOduration=2.892169146 podStartE2EDuration="6.94912878s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.056959634s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:26.947898071 +0000 UTC m=+2591.395728548" watchObservedRunningTime="2026-09-17 18:17:26.94912878 +0000 UTC m=+2591.396959305"
Sep 17 18:17:27.933810 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:27.933783    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:33.944575 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:33.944503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:17:49.918831 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:49.918779    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:17:59.875292 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:17:59.875262    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:18:03.948054 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:03.948027    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:18:07.296658 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:07.296623    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"]
Sep 17 18:18:07.297121 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:07.296851    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kserve-container" containerID="cri-o://f0a94ef2b5ee8fc576d966b219aa7e1fde904897eca94e75ed41eb7f18a1cda4" gracePeriod=30
Sep 17 18:18:07.297121 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:07.296879    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kube-rbac-proxy" containerID="cri-o://d07ec787ed483895013c8c93adf45d9001ea0fb2e12de609fba60028ad23ea44" gracePeriod=30
Sep 17 18:18:08.122913 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:08.122877    2562 generic.go:356] "Generic (PLEG): container finished" podID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerID="d07ec787ed483895013c8c93adf45d9001ea0fb2e12de609fba60028ad23ea44" exitCode=2
Sep 17 18:18:08.122913 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:08.122914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerDied","Data":"d07ec787ed483895013c8c93adf45d9001ea0fb2e12de609fba60028ad23ea44"}
Sep 17 18:18:09.867809 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:09.867770    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 18:18:10.868482 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:10.868450    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"]
Sep 17 18:18:10.868877 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:10.868714    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kserve-container" containerID="cri-o://32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795" gracePeriod=30
Sep 17 18:18:10.868877 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:10.868802    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kube-rbac-proxy" containerID="cri-o://374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284" gracePeriod=30
Sep 17 18:18:11.141856 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.141759    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerID="374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284" exitCode=2
Sep 17 18:18:11.142023 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.141835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerDied","Data":"374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284"}
Sep 17 18:18:11.812314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.812291    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:18:11.914879 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.914816    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-cabundle-cert\") pod \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") "
Sep 17 18:18:11.914879 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.914856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-rbac-proxy-sar-config\") pod \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") "
Sep 17 18:18:11.915369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.914913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrg8v\" (UniqueName: \"kubernetes.io/projected/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-api-access-vrg8v\") pod \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") "
Sep 17 18:18:11.915369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.914938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kserve-provision-location\") pod \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") "
Sep 17 18:18:11.915369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.914963    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-proxy-tls\") pod \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\" (UID: \"0e0ba285-3160-4ac7-bdff-b28d646dd9bb\") "
Sep 17 18:18:11.915369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.915123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e0ba285-3160-4ac7-bdff-b28d646dd9bb" (UID: "0e0ba285-3160-4ac7-bdff-b28d646dd9bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:11.915369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.915283    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e0ba285-3160-4ac7-bdff-b28d646dd9bb" (UID: "0e0ba285-3160-4ac7-bdff-b28d646dd9bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:11.915369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.915296    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e0ba285-3160-4ac7-bdff-b28d646dd9bb" (UID: "0e0ba285-3160-4ac7-bdff-b28d646dd9bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:11.917139 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.917111    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e0ba285-3160-4ac7-bdff-b28d646dd9bb" (UID: "0e0ba285-3160-4ac7-bdff-b28d646dd9bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:11.917200 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:11.917169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-api-access-vrg8v" (OuterVolumeSpecName: "kube-api-access-vrg8v") pod "0e0ba285-3160-4ac7-bdff-b28d646dd9bb" (UID: "0e0ba285-3160-4ac7-bdff-b28d646dd9bb"). InnerVolumeSpecName "kube-api-access-vrg8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:12.015717 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.015694    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrg8v\" (UniqueName: \"kubernetes.io/projected/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-api-access-vrg8v\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:12.015717 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.015714    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:12.015845 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.015723    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:12.015845 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.015734    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:12.015845 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.015743    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e0ba285-3160-4ac7-bdff-b28d646dd9bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:12.148565 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.148533    2562 generic.go:356] "Generic (PLEG): container finished" podID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerID="32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795" exitCode=0
Sep 17 18:18:12.148704 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.148589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerDied","Data":"32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795"}
Sep 17 18:18:12.148704 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.148645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx" event={"ID":"0e0ba285-3160-4ac7-bdff-b28d646dd9bb","Type":"ContainerDied","Data":"a68abe7783d0891d4b10941eeadfa23ca658349e13247c6403d769d710a2b41b"}
Sep 17 18:18:12.148704 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.148594    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"
Sep 17 18:18:12.148809 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.148673    2562 scope.go:117] "RemoveContainer" containerID="374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284"
Sep 17 18:18:12.157604 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.157585    2562 scope.go:117] "RemoveContainer" containerID="32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795"
Sep 17 18:18:12.165020 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.165003    2562 scope.go:117] "RemoveContainer" containerID="f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4"
Sep 17 18:18:12.171321 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.171299    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"]
Sep 17 18:18:12.172849 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.172830    2562 scope.go:117] "RemoveContainer" containerID="374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284"
Sep 17 18:18:12.173113 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:18:12.173093    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284\": container with ID starting with 374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284 not found: ID does not exist" containerID="374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284"
Sep 17 18:18:12.173169 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.173121    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284"} err="failed to get container status \"374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284\": rpc error: code = NotFound desc = could not find container \"374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284\": container with ID starting with 374eb442548d24649a313757ae6cfe6804047087505634527e0bbd2d4b00e284 not found: ID does not exist"
Sep 17 18:18:12.173169 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.173136    2562 scope.go:117] "RemoveContainer" containerID="32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795"
Sep 17 18:18:12.173452 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:18:12.173431    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795\": container with ID starting with 32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795 not found: ID does not exist" containerID="32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795"
Sep 17 18:18:12.173537 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.173456    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795"} err="failed to get container status \"32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795\": rpc error: code = NotFound desc = could not find container \"32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795\": container with ID starting with 32d8f3bfcf4d4b195307f6ea7e73430ce203abc5868ecb6de5c2bb1b1643b795 not found: ID does not exist"
Sep 17 18:18:12.173537 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.173471    2562 scope.go:117] "RemoveContainer" containerID="f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4"
Sep 17 18:18:12.173714 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:18:12.173696    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4\": container with ID starting with f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4 not found: ID does not exist" containerID="f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4"
Sep 17 18:18:12.173758 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.173719    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4"} err="failed to get container status \"f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4\": rpc error: code = NotFound desc = could not find container \"f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4\": container with ID starting with f1d0a22a0d017314ec17fb08051ccfaf1dd088b8b7a515cdfaacefd2762c11b4 not found: ID does not exist"
Sep 17 18:18:12.173947 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:12.173931    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-k9qzx"]
Sep 17 18:18:13.726715 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:13.726681    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" path="/var/lib/kubelet/pods/0e0ba285-3160-4ac7-bdff-b28d646dd9bb/volumes"
Sep 17 18:18:14.160725 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.160696    2562 generic.go:356] "Generic (PLEG): container finished" podID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerID="f0a94ef2b5ee8fc576d966b219aa7e1fde904897eca94e75ed41eb7f18a1cda4" exitCode=0
Sep 17 18:18:14.160881 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.160771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerDied","Data":"f0a94ef2b5ee8fc576d966b219aa7e1fde904897eca94e75ed41eb7f18a1cda4"}
Sep 17 18:18:14.243983 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.243962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:18:14.333398 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82125b9d-5466-4a1d-a8a0-e665baa0fa48-proxy-tls\") pod \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") "
Sep 17 18:18:14.333550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-cabundle-cert\") pod \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") "
Sep 17 18:18:14.333550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z24bx\" (UniqueName: \"kubernetes.io/projected/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-api-access-z24bx\") pod \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") "
Sep 17 18:18:14.333550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333491    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kserve-provision-location\") pod \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") "
Sep 17 18:18:14.333550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333519    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-rbac-proxy-sar-config\") pod \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\" (UID: \"82125b9d-5466-4a1d-a8a0-e665baa0fa48\") "
Sep 17 18:18:14.333840 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82125b9d-5466-4a1d-a8a0-e665baa0fa48" (UID: "82125b9d-5466-4a1d-a8a0-e665baa0fa48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:14.333901 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333829    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82125b9d-5466-4a1d-a8a0-e665baa0fa48" (UID: "82125b9d-5466-4a1d-a8a0-e665baa0fa48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:14.333901 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.333883    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82125b9d-5466-4a1d-a8a0-e665baa0fa48" (UID: "82125b9d-5466-4a1d-a8a0-e665baa0fa48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:14.335651 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.335621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-api-access-z24bx" (OuterVolumeSpecName: "kube-api-access-z24bx") pod "82125b9d-5466-4a1d-a8a0-e665baa0fa48" (UID: "82125b9d-5466-4a1d-a8a0-e665baa0fa48"). InnerVolumeSpecName "kube-api-access-z24bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:14.335739 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.335695    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82125b9d-5466-4a1d-a8a0-e665baa0fa48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82125b9d-5466-4a1d-a8a0-e665baa0fa48" (UID: "82125b9d-5466-4a1d-a8a0-e665baa0fa48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:14.434312 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.434273    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:14.434312 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.434305    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24bx\" (UniqueName: \"kubernetes.io/projected/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-api-access-z24bx\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:14.434312 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.434314    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:14.434517 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.434347    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82125b9d-5466-4a1d-a8a0-e665baa0fa48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:14.434517 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:14.434361    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82125b9d-5466-4a1d-a8a0-e665baa0fa48-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:18:15.167230 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.167199    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"
Sep 17 18:18:15.167230 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.167215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9" event={"ID":"82125b9d-5466-4a1d-a8a0-e665baa0fa48","Type":"ContainerDied","Data":"72ce1ae1b0f69b3cc616b95dd58afe52f09fc867a67d294b0d1578ea4f828080"}
Sep 17 18:18:15.167752 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.167283    2562 scope.go:117] "RemoveContainer" containerID="d07ec787ed483895013c8c93adf45d9001ea0fb2e12de609fba60028ad23ea44"
Sep 17 18:18:15.176018 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.176002    2562 scope.go:117] "RemoveContainer" containerID="f0a94ef2b5ee8fc576d966b219aa7e1fde904897eca94e75ed41eb7f18a1cda4"
Sep 17 18:18:15.183800 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.183781    2562 scope.go:117] "RemoveContainer" containerID="5113ca25201ccb168cb6c8e145e3ca3d00db1b8c6d4943cd5e8019cc4c0ad1e3"
Sep 17 18:18:15.192428 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.192409    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"]
Sep 17 18:18:15.201133 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.201115    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-qshh9"]
Sep 17 18:18:15.731235 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:15.731206    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" path="/var/lib/kubelet/pods/82125b9d-5466-4a1d-a8a0-e665baa0fa48/volumes"
Sep 17 18:18:51.365047 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365016    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"]
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365342    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kube-rbac-proxy"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365354    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kube-rbac-proxy"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365362    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="storage-initializer"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365368    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="storage-initializer"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365384    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="storage-initializer"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365390    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="storage-initializer"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365401    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kube-rbac-proxy"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365406    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kube-rbac-proxy"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365413    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kserve-container"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365419    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kserve-container"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365427    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kserve-container"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365432    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kserve-container"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365486    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kube-rbac-proxy"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365494    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e0ba285-3160-4ac7-bdff-b28d646dd9bb" containerName="kserve-container"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365501    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kserve-container"
Sep 17 18:18:51.365526 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.365507    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="82125b9d-5466-4a1d-a8a0-e665baa0fa48" containerName="kube-rbac-proxy"
Sep 17 18:18:51.368611 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.368595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.371249 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.371225    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 18:18:51.371400 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.371307    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:18:51.371400 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.371310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:18:51.371517 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.371361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:51.371614 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.371534    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:18:51.372426 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.372404    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:18:51.372541 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.372464    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:51.380404 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.380385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"]
Sep 17 18:18:51.433014 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.432982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhvt\" (UniqueName: \"kubernetes.io/projected/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-api-access-xhhvt\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.433170 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.433056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e598dd-37c9-47d4-a430-5ce2b506b413-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.433170 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.433100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.433170 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.433130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.433512 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.433162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.534580 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.534549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e598dd-37c9-47d4-a430-5ce2b506b413-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.534762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.534598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.534762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.534626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.534762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.534649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.534762 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.534696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhvt\" (UniqueName: \"kubernetes.io/projected/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-api-access-xhhvt\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.535042 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:18:51.534780    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 18:18:51.535042 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:18:51.534863    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls podName:f2e598dd-37c9-47d4-a430-5ce2b506b413 nodeName:}" failed. No retries permitted until 2026-09-17 18:18:52.03484234 +0000 UTC m=+2676.482672779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" (UID: "f2e598dd-37c9-47d4-a430-5ce2b506b413") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 18:18:51.535147 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.535078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e598dd-37c9-47d4-a430-5ce2b506b413-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.535412 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.535394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.535457 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.535409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:51.544084 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:51.544062    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhvt\" (UniqueName: \"kubernetes.io/projected/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-api-access-xhhvt\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:52.038606 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:52.038567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:52.040829 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:52.040807    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:52.279439 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:52.279401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:52.552052 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:52.552022    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"]
Sep 17 18:18:52.554097 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:18:52.554071    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e598dd_37c9_47d4_a430_5ce2b506b413.slice/crio-665a368f5dd26124d19bdb45ac7735e14732fe4757bc46f6c4850144f83aacbc WatchSource:0}: Error finding container 665a368f5dd26124d19bdb45ac7735e14732fe4757bc46f6c4850144f83aacbc: Status 404 returned error can't find the container with id 665a368f5dd26124d19bdb45ac7735e14732fe4757bc46f6c4850144f83aacbc
Sep 17 18:18:53.335026 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:53.334991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerStarted","Data":"cdeb6f45c0572ea0e48a6a20116c53f6d0fe70d1cb5621e4cca827471dbb0ebb"}
Sep 17 18:18:53.335206 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:53.335056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerStarted","Data":"665a368f5dd26124d19bdb45ac7735e14732fe4757bc46f6c4850144f83aacbc"}
Sep 17 18:18:57.355408 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:57.355377    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerID="cdeb6f45c0572ea0e48a6a20116c53f6d0fe70d1cb5621e4cca827471dbb0ebb" exitCode=0
Sep 17 18:18:57.355975 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:57.355418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerDied","Data":"cdeb6f45c0572ea0e48a6a20116c53f6d0fe70d1cb5621e4cca827471dbb0ebb"}
Sep 17 18:18:58.362550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:58.362510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerStarted","Data":"beacb67172338c71b8205338a291b5210e2ac54660ac42f772568263d3c6fb6e"}
Sep 17 18:18:58.362959 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:58.362578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerStarted","Data":"8716eafbf4444f45db905c9bcc33797f71390fa177607a525feb202a3aa0bc24"}
Sep 17 18:18:58.362959 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:58.362818    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:58.362959 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:58.362878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:18:58.385459 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:18:58.385406    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" podStartSLOduration=3.3228931729999998 podStartE2EDuration="7.385392063s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.06249889s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:58.384204248 +0000 UTC m=+2682.832034724" watchObservedRunningTime="2026-09-17 18:18:58.385392063 +0000 UTC m=+2682.833222517"
Sep 17 18:19:04.373980 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:04.373949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:19:17.888235 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:17.888198    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:19:17.894979 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:17.894957    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:19:34.378148 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:34.378115    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:19:41.398506 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:41.398473    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"]
Sep 17 18:19:41.398961 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:41.398752    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kube-rbac-proxy" containerID="cri-o://beacb67172338c71b8205338a291b5210e2ac54660ac42f772568263d3c6fb6e" gracePeriod=30
Sep 17 18:19:41.398961 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:41.398728    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kserve-container" containerID="cri-o://8716eafbf4444f45db905c9bcc33797f71390fa177607a525feb202a3aa0bc24" gracePeriod=30
Sep 17 18:19:41.567211 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:41.567175    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerID="beacb67172338c71b8205338a291b5210e2ac54660ac42f772568263d3c6fb6e" exitCode=2
Sep 17 18:19:41.567430 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:41.567225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerDied","Data":"beacb67172338c71b8205338a291b5210e2ac54660ac42f772568263d3c6fb6e"}
Sep 17 18:19:42.574797 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.574756    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerID="8716eafbf4444f45db905c9bcc33797f71390fa177607a525feb202a3aa0bc24" exitCode=0
Sep 17 18:19:42.575166 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.574821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerDied","Data":"8716eafbf4444f45db905c9bcc33797f71390fa177607a525feb202a3aa0bc24"}
Sep 17 18:19:42.660395 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.660368    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:19:42.775661 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.775567    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhhvt\" (UniqueName: \"kubernetes.io/projected/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-api-access-xhhvt\") pod \"f2e598dd-37c9-47d4-a430-5ce2b506b413\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") "
Sep 17 18:19:42.775661 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.775617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-cabundle-cert\") pod \"f2e598dd-37c9-47d4-a430-5ce2b506b413\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") "
Sep 17 18:19:42.775661 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.775638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-rbac-proxy-sar-config\") pod \"f2e598dd-37c9-47d4-a430-5ce2b506b413\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") "
Sep 17 18:19:42.775962 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.775764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e598dd-37c9-47d4-a430-5ce2b506b413-kserve-provision-location\") pod \"f2e598dd-37c9-47d4-a430-5ce2b506b413\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") "
Sep 17 18:19:42.775962 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.775896    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls\") pod \"f2e598dd-37c9-47d4-a430-5ce2b506b413\" (UID: \"f2e598dd-37c9-47d4-a430-5ce2b506b413\") "
Sep 17 18:19:42.776069 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.776034    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2e598dd-37c9-47d4-a430-5ce2b506b413" (UID: "f2e598dd-37c9-47d4-a430-5ce2b506b413"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:42.776134 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.776066    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2e598dd-37c9-47d4-a430-5ce2b506b413" (UID: "f2e598dd-37c9-47d4-a430-5ce2b506b413"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:42.776134 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.776112    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e598dd-37c9-47d4-a430-5ce2b506b413-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2e598dd-37c9-47d4-a430-5ce2b506b413" (UID: "f2e598dd-37c9-47d4-a430-5ce2b506b413"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:42.776277 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.776258    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:19:42.776382 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.776282    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:19:42.776382 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.776296    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2e598dd-37c9-47d4-a430-5ce2b506b413-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:19:42.777947 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.777925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-api-access-xhhvt" (OuterVolumeSpecName: "kube-api-access-xhhvt") pod "f2e598dd-37c9-47d4-a430-5ce2b506b413" (UID: "f2e598dd-37c9-47d4-a430-5ce2b506b413"). InnerVolumeSpecName "kube-api-access-xhhvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:42.778053 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.777946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2e598dd-37c9-47d4-a430-5ce2b506b413" (UID: "f2e598dd-37c9-47d4-a430-5ce2b506b413"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:42.877052 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.877017    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2e598dd-37c9-47d4-a430-5ce2b506b413-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:19:42.877052 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:42.877048    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhhvt\" (UniqueName: \"kubernetes.io/projected/f2e598dd-37c9-47d4-a430-5ce2b506b413-kube-api-access-xhhvt\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:19:43.582363 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.582312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2" event={"ID":"f2e598dd-37c9-47d4-a430-5ce2b506b413","Type":"ContainerDied","Data":"665a368f5dd26124d19bdb45ac7735e14732fe4757bc46f6c4850144f83aacbc"}
Sep 17 18:19:43.582783 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.582371    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"
Sep 17 18:19:43.582783 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.582424    2562 scope.go:117] "RemoveContainer" containerID="beacb67172338c71b8205338a291b5210e2ac54660ac42f772568263d3c6fb6e"
Sep 17 18:19:43.591319 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.591297    2562 scope.go:117] "RemoveContainer" containerID="8716eafbf4444f45db905c9bcc33797f71390fa177607a525feb202a3aa0bc24"
Sep 17 18:19:43.599312 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.599290    2562 scope.go:117] "RemoveContainer" containerID="cdeb6f45c0572ea0e48a6a20116c53f6d0fe70d1cb5621e4cca827471dbb0ebb"
Sep 17 18:19:43.605520 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.605383    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"]
Sep 17 18:19:43.611396 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.611374    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zbfp2"]
Sep 17 18:19:43.727341 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:43.727303    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" path="/var/lib/kubelet/pods/f2e598dd-37c9-47d4-a430-5ce2b506b413/volumes"
Sep 17 18:19:46.610339 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610286    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"]
Sep 17 18:19:46.610816 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610797    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="storage-initializer"
Sep 17 18:19:46.610888 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610821    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="storage-initializer"
Sep 17 18:19:46.610888 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610851    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kube-rbac-proxy"
Sep 17 18:19:46.610888 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610861    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kube-rbac-proxy"
Sep 17 18:19:46.610888 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610885    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kserve-container"
Sep 17 18:19:46.611089 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610894    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kserve-container"
Sep 17 18:19:46.611089 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610978    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kserve-container"
Sep 17 18:19:46.611089 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.610991    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e598dd-37c9-47d4-a430-5ce2b506b413" containerName="kube-rbac-proxy"
Sep 17 18:19:46.615839 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.615819    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.618301 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618281    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 18:19:46.618450 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618308    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:19:46.618450 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618308    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:19:46.618578 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618524    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:19:46.618640 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618592    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 18:19:46.618714 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618696    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:19:46.618807 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.618790    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:19:46.625441 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.625415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"]
Sep 17 18:19:46.709171 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.709136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755c1415-f90d-44d7-adb7-44d85b189086-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.709171 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.709173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6smm\" (UniqueName: \"kubernetes.io/projected/755c1415-f90d-44d7-adb7-44d85b189086-kube-api-access-q6smm\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.709434 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.709205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.709434 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.709283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755c1415-f90d-44d7-adb7-44d85b189086-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.709434 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.709321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810012 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.809976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755c1415-f90d-44d7-adb7-44d85b189086-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810012 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6smm\" (UniqueName: \"kubernetes.io/projected/755c1415-f90d-44d7-adb7-44d85b189086-kube-api-access-q6smm\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755c1415-f90d-44d7-adb7-44d85b189086-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810584 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755c1415-f90d-44d7-adb7-44d85b189086-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810803 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.810848 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.810789    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.812557 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.812536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755c1415-f90d-44d7-adb7-44d85b189086-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.817469 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.817449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6smm\" (UniqueName: \"kubernetes.io/projected/755c1415-f90d-44d7-adb7-44d85b189086-kube-api-access-q6smm\") pod \"isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:46.928869 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:46.928780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:47.204926 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:47.204898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"]
Sep 17 18:19:47.207310 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:19:47.207279    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755c1415_f90d_44d7_adb7_44d85b189086.slice/crio-dd2b9d990cb2aca9d2de9ad077016840fd811824c7a77d2e92a72238605aa05f WatchSource:0}: Error finding container dd2b9d990cb2aca9d2de9ad077016840fd811824c7a77d2e92a72238605aa05f: Status 404 returned error can't find the container with id dd2b9d990cb2aca9d2de9ad077016840fd811824c7a77d2e92a72238605aa05f
Sep 17 18:19:47.603404 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:47.603367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerStarted","Data":"e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a"}
Sep 17 18:19:47.603614 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:47.603449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerStarted","Data":"dd2b9d990cb2aca9d2de9ad077016840fd811824c7a77d2e92a72238605aa05f"}
Sep 17 18:19:51.622371 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:51.622323    2562 generic.go:356] "Generic (PLEG): container finished" podID="755c1415-f90d-44d7-adb7-44d85b189086" containerID="e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a" exitCode=0
Sep 17 18:19:51.622748 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:51.622378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerDied","Data":"e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a"}
Sep 17 18:19:51.683746 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:51.683725    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:19:52.631695 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:52.631656    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerStarted","Data":"ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152"}
Sep 17 18:19:55.649286 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:55.649245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerStarted","Data":"e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c"}
Sep 17 18:19:55.649670 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:55.649373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerStarted","Data":"2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302"}
Sep 17 18:19:55.649670 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:55.649423    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:55.649670 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:55.649558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:19:55.672576 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:55.672531    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podStartSLOduration=2.637796154 podStartE2EDuration="9.6725182s" totalImagesPullingTime="3.307833232s" totalInitContainerRuntime="3.726888814s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:55.67183902 +0000 UTC m=+2740.119669495" watchObservedRunningTime="2026-09-17 18:19:55.6725182 +0000 UTC m=+2740.120348656"
Sep 17 18:19:56.654724 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:19:56.654693    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:20:02.665368 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:20:02.665320    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:20:22.666657 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:20:22.666619    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:20:26.929245 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:20:26.929193    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:20:32.698963 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:20:32.698880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:21:02.699976 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:02.699941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:21:06.665694 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:06.665655    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"]
Sep 17 18:21:06.666162 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:06.665989    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container" containerID="cri-o://ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152" gracePeriod=30
Sep 17 18:21:06.666162 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:06.666025    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-agent" containerID="cri-o://2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302" gracePeriod=30
Sep 17 18:21:06.666162 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:06.666026    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" containerID="cri-o://e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c" gracePeriod=30
Sep 17 18:21:06.974681 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:06.974580    2562 generic.go:356] "Generic (PLEG): container finished" podID="755c1415-f90d-44d7-adb7-44d85b189086" containerID="e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c" exitCode=2
Sep 17 18:21:06.974681 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:06.974656    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerDied","Data":"e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c"}
Sep 17 18:21:07.660479 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:07.660442    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:21:08.986164 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:08.986079    2562 generic.go:356] "Generic (PLEG): container finished" podID="755c1415-f90d-44d7-adb7-44d85b189086" containerID="ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152" exitCode=0
Sep 17 18:21:08.986164 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:08.986118    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerDied","Data":"ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152"}
Sep 17 18:21:12.660748 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:12.660703    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:21:12.666084 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:12.666047    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:21:17.661091 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:17.661053    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:21:17.661480 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:17.661189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:21:22.660388 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:22.660317    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:21:22.666781 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:22.666758    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:21:27.660108 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:27.660062    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:21:32.665349 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:32.665274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:21:32.665984 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:32.665957    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:21:32.666101 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:32.666088    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:21:36.857575 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.857553    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:21:36.918376 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918314    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-cabundle-cert\") pod \"755c1415-f90d-44d7-adb7-44d85b189086\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") "
Sep 17 18:21:36.918550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918409    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755c1415-f90d-44d7-adb7-44d85b189086-proxy-tls\") pod \"755c1415-f90d-44d7-adb7-44d85b189086\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") "
Sep 17 18:21:36.918550 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918435    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-kube-rbac-proxy-sar-config\") pod \"755c1415-f90d-44d7-adb7-44d85b189086\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") "
Sep 17 18:21:36.918676 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918555    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6smm\" (UniqueName: \"kubernetes.io/projected/755c1415-f90d-44d7-adb7-44d85b189086-kube-api-access-q6smm\") pod \"755c1415-f90d-44d7-adb7-44d85b189086\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") "
Sep 17 18:21:36.918676 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755c1415-f90d-44d7-adb7-44d85b189086-kserve-provision-location\") pod \"755c1415-f90d-44d7-adb7-44d85b189086\" (UID: \"755c1415-f90d-44d7-adb7-44d85b189086\") "
Sep 17 18:21:36.918789 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918754    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "755c1415-f90d-44d7-adb7-44d85b189086" (UID: "755c1415-f90d-44d7-adb7-44d85b189086"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:36.918850 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918828    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:21:36.918961 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "755c1415-f90d-44d7-adb7-44d85b189086" (UID: "755c1415-f90d-44d7-adb7-44d85b189086"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:36.918961 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.918949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755c1415-f90d-44d7-adb7-44d85b189086-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "755c1415-f90d-44d7-adb7-44d85b189086" (UID: "755c1415-f90d-44d7-adb7-44d85b189086"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:36.920498 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.920457    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755c1415-f90d-44d7-adb7-44d85b189086-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "755c1415-f90d-44d7-adb7-44d85b189086" (UID: "755c1415-f90d-44d7-adb7-44d85b189086"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:36.920617 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:36.920597    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755c1415-f90d-44d7-adb7-44d85b189086-kube-api-access-q6smm" (OuterVolumeSpecName: "kube-api-access-q6smm") pod "755c1415-f90d-44d7-adb7-44d85b189086" (UID: "755c1415-f90d-44d7-adb7-44d85b189086"). InnerVolumeSpecName "kube-api-access-q6smm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:37.020130 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.020102    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6smm\" (UniqueName: \"kubernetes.io/projected/755c1415-f90d-44d7-adb7-44d85b189086-kube-api-access-q6smm\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:21:37.020130 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.020125    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755c1415-f90d-44d7-adb7-44d85b189086-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:21:37.020130 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.020135    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755c1415-f90d-44d7-adb7-44d85b189086-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:21:37.020357 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.020145    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755c1415-f90d-44d7-adb7-44d85b189086-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:21:37.122875 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.122846    2562 generic.go:356] "Generic (PLEG): container finished" podID="755c1415-f90d-44d7-adb7-44d85b189086" containerID="2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302" exitCode=0
Sep 17 18:21:37.123040 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.122910    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"
Sep 17 18:21:37.123040 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.122927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerDied","Data":"2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302"}
Sep 17 18:21:37.123040 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.123006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw" event={"ID":"755c1415-f90d-44d7-adb7-44d85b189086","Type":"ContainerDied","Data":"dd2b9d990cb2aca9d2de9ad077016840fd811824c7a77d2e92a72238605aa05f"}
Sep 17 18:21:37.123206 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.123050    2562 scope.go:117] "RemoveContainer" containerID="e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c"
Sep 17 18:21:37.132017 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.132000    2562 scope.go:117] "RemoveContainer" containerID="2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302"
Sep 17 18:21:37.139662 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.139647    2562 scope.go:117] "RemoveContainer" containerID="ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152"
Sep 17 18:21:37.146992 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.146947    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"]
Sep 17 18:21:37.148168 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.148151    2562 scope.go:117] "RemoveContainer" containerID="e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a"
Sep 17 18:21:37.148981 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.148960    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-8584bb4bfb-qxtbw"]
Sep 17 18:21:37.155815 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.155799    2562 scope.go:117] "RemoveContainer" containerID="e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c"
Sep 17 18:21:37.156080 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:21:37.156061    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c\": container with ID starting with e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c not found: ID does not exist" containerID="e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c"
Sep 17 18:21:37.156126 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156088    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c"} err="failed to get container status \"e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c\": rpc error: code = NotFound desc = could not find container \"e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c\": container with ID starting with e50bc65b66d3f0b9ea494b8e8b57a66f3ec725f0c1d56df5f85cab4d079a047c not found: ID does not exist"
Sep 17 18:21:37.156126 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156105    2562 scope.go:117] "RemoveContainer" containerID="2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302"
Sep 17 18:21:37.156356 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:21:37.156324    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302\": container with ID starting with 2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302 not found: ID does not exist" containerID="2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302"
Sep 17 18:21:37.156407 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156362    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302"} err="failed to get container status \"2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302\": rpc error: code = NotFound desc = could not find container \"2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302\": container with ID starting with 2a919c95f3b0377b1869296f618fc5bde18830d7597b4c2557bd02cac22cc302 not found: ID does not exist"
Sep 17 18:21:37.156407 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156379    2562 scope.go:117] "RemoveContainer" containerID="ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152"
Sep 17 18:21:37.156605 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:21:37.156589    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152\": container with ID starting with ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152 not found: ID does not exist" containerID="ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152"
Sep 17 18:21:37.156649 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156609    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152"} err="failed to get container status \"ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152\": rpc error: code = NotFound desc = could not find container \"ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152\": container with ID starting with ed10721ca7e6fe514a73a294a6a8f793b3b4381c1256106ed58c042c2ad61152 not found: ID does not exist"
Sep 17 18:21:37.156649 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156622    2562 scope.go:117] "RemoveContainer" containerID="e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a"
Sep 17 18:21:37.156823 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:21:37.156808    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a\": container with ID starting with e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a not found: ID does not exist" containerID="e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a"
Sep 17 18:21:37.156865 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.156827    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a"} err="failed to get container status \"e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a\": rpc error: code = NotFound desc = could not find container \"e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a\": container with ID starting with e0b571a89c08a1ab8f8c960a8d4c73def5bbe1f7a675fe5914262cd023b5292a not found: ID does not exist"
Sep 17 18:21:37.727805 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:21:37.727772    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="755c1415-f90d-44d7-adb7-44d85b189086" path="/var/lib/kubelet/pods/755c1415-f90d-44d7-adb7-44d85b189086/volumes"
Sep 17 18:22:28.598857 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.598813    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"]
Sep 17 18:22:28.599606 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599565    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="storage-initializer"
Sep 17 18:22:28.599606 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599593    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="storage-initializer"
Sep 17 18:22:28.599761 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599640    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-agent"
Sep 17 18:22:28.599761 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599651    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-agent"
Sep 17 18:22:28.599761 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599682    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy"
Sep 17 18:22:28.599761 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599692    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy"
Sep 17 18:22:28.599761 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599723    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container"
Sep 17 18:22:28.599761 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599733    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container"
Sep 17 18:22:28.600032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599883    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-container"
Sep 17 18:22:28.600032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599908    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kserve-agent"
Sep 17 18:22:28.600032 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.599918    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="755c1415-f90d-44d7-adb7-44d85b189086" containerName="kube-rbac-proxy"
Sep 17 18:22:28.603976 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.603955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.614197 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.614175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:22:28.614387 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.614203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:22:28.614795 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.614759    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:22:28.614910 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.614801    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:22:28.615223 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.615178    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:28.615866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.615849    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:28.617486 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.617466    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"]
Sep 17 18:22:28.622238 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.622220    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 18:22:28.749026 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.748993    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f893d4-20bd-4c6d-bd93-67c035f314be-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.749217 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.749052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f893d4-20bd-4c6d-bd93-67c035f314be-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.749217 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.749138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvcc6\" (UniqueName: \"kubernetes.io/projected/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-api-access-dvcc6\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.749217 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.749187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.749353 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.749226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.849729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.849646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f893d4-20bd-4c6d-bd93-67c035f314be-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.849729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.849684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvcc6\" (UniqueName: \"kubernetes.io/projected/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-api-access-dvcc6\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.849729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.849710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.849729 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.849729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.850067 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.849767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f893d4-20bd-4c6d-bd93-67c035f314be-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.850154 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.850129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f893d4-20bd-4c6d-bd93-67c035f314be-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.850448 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.850422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.850542 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.850478    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.852307 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.852287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f893d4-20bd-4c6d-bd93-67c035f314be-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.863896 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.863873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvcc6\" (UniqueName: \"kubernetes.io/projected/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-api-access-dvcc6\") pod \"isvc-paddle-runtime-predictor-58b99fff76-rfmdg\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:28.916772 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:28.916732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:29.207481 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:29.207452    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"]
Sep 17 18:22:29.210176 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:22:29.210135    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f893d4_20bd_4c6d_bd93_67c035f314be.slice/crio-719f38f745deb3b9394b87fffdddeb698803b6c7715cc25897e8be3d1b3c35be WatchSource:0}: Error finding container 719f38f745deb3b9394b87fffdddeb698803b6c7715cc25897e8be3d1b3c35be: Status 404 returned error can't find the container with id 719f38f745deb3b9394b87fffdddeb698803b6c7715cc25897e8be3d1b3c35be
Sep 17 18:22:29.352316 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:29.352277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerStarted","Data":"97c8f2e61742ef1bfb2a29a4d1d046b143fc8912d1f3550079407735e35b1087"}
Sep 17 18:22:29.352525 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:29.352386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerStarted","Data":"719f38f745deb3b9394b87fffdddeb698803b6c7715cc25897e8be3d1b3c35be"}
Sep 17 18:22:34.377838 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:34.377802    2562 generic.go:356] "Generic (PLEG): container finished" podID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerID="97c8f2e61742ef1bfb2a29a4d1d046b143fc8912d1f3550079407735e35b1087" exitCode=0
Sep 17 18:22:34.378244 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:34.377886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerDied","Data":"97c8f2e61742ef1bfb2a29a4d1d046b143fc8912d1f3550079407735e35b1087"}
Sep 17 18:22:46.444925 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:46.444888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerStarted","Data":"caefbf4111663a4eba8a3b1d15881802a52a8194ea6335f98e1d5e3b480dee7f"}
Sep 17 18:22:46.445369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:46.444951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerStarted","Data":"366c38b383d683cea01defe6bb449bc27ee485f155db356cddf4f8c17641023a"}
Sep 17 18:22:46.445369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:46.445232    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:46.445369 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:46.445280    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:46.446390 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:46.446367    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:22:46.476570 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:46.476524    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podStartSLOduration=2.265478343 podStartE2EDuration="18.476510907s" totalImagesPullingTime="11.25533519s" totalInitContainerRuntime="4.955697374s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:46.474457899 +0000 UTC m=+2910.922288375" watchObservedRunningTime="2026-09-17 18:22:46.476510907 +0000 UTC m=+2910.924341433"
Sep 17 18:22:47.450557 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:47.450514    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:22:52.454542 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:52.454513    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:22:52.455097 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:22:52.455071    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:23:02.455608 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:02.455565    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:23:12.455248 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:12.455210    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:23:22.455769 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:22.455732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:23:32.456024 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:32.455949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:23:39.968834 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:39.968799    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"]
Sep 17 18:23:39.969219 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:39.969050    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" containerID="cri-o://366c38b383d683cea01defe6bb449bc27ee485f155db356cddf4f8c17641023a" gracePeriod=30
Sep 17 18:23:39.969219 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:39.969100    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kube-rbac-proxy" containerID="cri-o://caefbf4111663a4eba8a3b1d15881802a52a8194ea6335f98e1d5e3b480dee7f" gracePeriod=30
Sep 17 18:23:40.693400 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:40.693362    2562 generic.go:356] "Generic (PLEG): container finished" podID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerID="caefbf4111663a4eba8a3b1d15881802a52a8194ea6335f98e1d5e3b480dee7f" exitCode=2
Sep 17 18:23:40.693584 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:40.693416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerDied","Data":"caefbf4111663a4eba8a3b1d15881802a52a8194ea6335f98e1d5e3b480dee7f"}
Sep 17 18:23:42.451139 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.451097    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 17 18:23:42.454995 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.454964    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:23:42.704129 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.704059    2562 generic.go:356] "Generic (PLEG): container finished" podID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerID="366c38b383d683cea01defe6bb449bc27ee485f155db356cddf4f8c17641023a" exitCode=0
Sep 17 18:23:42.704260 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.704130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerDied","Data":"366c38b383d683cea01defe6bb449bc27ee485f155db356cddf4f8c17641023a"}
Sep 17 18:23:42.704260 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.704184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg" event={"ID":"44f893d4-20bd-4c6d-bd93-67c035f314be","Type":"ContainerDied","Data":"719f38f745deb3b9394b87fffdddeb698803b6c7715cc25897e8be3d1b3c35be"}
Sep 17 18:23:42.704260 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.704205    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719f38f745deb3b9394b87fffdddeb698803b6c7715cc25897e8be3d1b3c35be"
Sep 17 18:23:42.714810 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.714789    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:23:42.770102 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-cabundle-cert\") pod \"44f893d4-20bd-4c6d-bd93-67c035f314be\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") "
Sep 17 18:23:42.770215 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770114    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-rbac-proxy-sar-config\") pod \"44f893d4-20bd-4c6d-bd93-67c035f314be\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") "
Sep 17 18:23:42.770215 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770172    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f893d4-20bd-4c6d-bd93-67c035f314be-kserve-provision-location\") pod \"44f893d4-20bd-4c6d-bd93-67c035f314be\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") "
Sep 17 18:23:42.770215 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvcc6\" (UniqueName: \"kubernetes.io/projected/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-api-access-dvcc6\") pod \"44f893d4-20bd-4c6d-bd93-67c035f314be\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") "
Sep 17 18:23:42.770418 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f893d4-20bd-4c6d-bd93-67c035f314be-proxy-tls\") pod \"44f893d4-20bd-4c6d-bd93-67c035f314be\" (UID: \"44f893d4-20bd-4c6d-bd93-67c035f314be\") "
Sep 17 18:23:42.770555 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44f893d4-20bd-4c6d-bd93-67c035f314be" (UID: "44f893d4-20bd-4c6d-bd93-67c035f314be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:42.770712 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.770540    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44f893d4-20bd-4c6d-bd93-67c035f314be" (UID: "44f893d4-20bd-4c6d-bd93-67c035f314be"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:42.772342 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.772288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-api-access-dvcc6" (OuterVolumeSpecName: "kube-api-access-dvcc6") pod "44f893d4-20bd-4c6d-bd93-67c035f314be" (UID: "44f893d4-20bd-4c6d-bd93-67c035f314be"). InnerVolumeSpecName "kube-api-access-dvcc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:42.772445 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.772384    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f893d4-20bd-4c6d-bd93-67c035f314be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44f893d4-20bd-4c6d-bd93-67c035f314be" (UID: "44f893d4-20bd-4c6d-bd93-67c035f314be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:42.780165 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.780143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f893d4-20bd-4c6d-bd93-67c035f314be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44f893d4-20bd-4c6d-bd93-67c035f314be" (UID: "44f893d4-20bd-4c6d-bd93-67c035f314be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:42.870953 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.870934    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:23:42.870953 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.870952    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:23:42.871066 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.870961    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f893d4-20bd-4c6d-bd93-67c035f314be-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:23:42.871066 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.870969    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvcc6\" (UniqueName: \"kubernetes.io/projected/44f893d4-20bd-4c6d-bd93-67c035f314be-kube-api-access-dvcc6\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:23:42.871066 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:42.870977    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f893d4-20bd-4c6d-bd93-67c035f314be-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:23:43.708878 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:43.708843    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"
Sep 17 18:23:43.731364 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:43.731319    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"]
Sep 17 18:23:43.739189 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:43.739163    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-rfmdg"]
Sep 17 18:23:45.080198 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080163    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"]
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080498    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="storage-initializer"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080510    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="storage-initializer"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080524    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kube-rbac-proxy"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080529    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kube-rbac-proxy"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080553    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080607    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kserve-container"
Sep 17 18:23:45.080674 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.080619    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" containerName="kube-rbac-proxy"
Sep 17 18:23:45.082732 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.082714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.085758 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.085721    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 18:23:45.085912 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.085854    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:23:45.085912 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.085876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:23:45.086177 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.086161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:23:45.086893 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.086870    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:23:45.087011 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.086909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:23:45.087011 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.086972    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:23:45.096139 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.096117    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"]
Sep 17 18:23:45.193724 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.193703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.193834 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.193738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.193834 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.193758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.193914 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.193839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.193914 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.193877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-api-access-szwhq\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294209 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294307 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294307 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294307 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294520 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-api-access-szwhq\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294631 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.294938 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.295051 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.294937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.296735 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.296715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.304358 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.304313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-api-access-szwhq\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.394006 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.393949    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:45.670259 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.670235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"]
Sep 17 18:23:45.672238 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:23:45.672205    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307eaa8f_edf7_4b6f_b2bf_5dff9fc4a751.slice/crio-d74f705b874e41c3dfc425f16a73429b1b1d15b99aac825146c1f2d083973819 WatchSource:0}: Error finding container d74f705b874e41c3dfc425f16a73429b1b1d15b99aac825146c1f2d083973819: Status 404 returned error can't find the container with id d74f705b874e41c3dfc425f16a73429b1b1d15b99aac825146c1f2d083973819
Sep 17 18:23:45.717923 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.717895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerStarted","Data":"d74f705b874e41c3dfc425f16a73429b1b1d15b99aac825146c1f2d083973819"}
Sep 17 18:23:45.728434 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:45.728409    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f893d4-20bd-4c6d-bd93-67c035f314be" path="/var/lib/kubelet/pods/44f893d4-20bd-4c6d-bd93-67c035f314be/volumes"
Sep 17 18:23:46.724291 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:46.724254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerStarted","Data":"9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744"}
Sep 17 18:23:50.742068 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:50.741993    2562 generic.go:356] "Generic (PLEG): container finished" podID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerID="9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744" exitCode=0
Sep 17 18:23:50.742433 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:50.742063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerDied","Data":"9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744"}
Sep 17 18:23:51.749042 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:51.749008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerStarted","Data":"1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa"}
Sep 17 18:23:51.749433 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:51.749066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerStarted","Data":"91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663"}
Sep 17 18:23:51.749433 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:51.749297    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:51.771314 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:51.771274    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podStartSLOduration=2.078784471 podStartE2EDuration="6.771261008s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.692476537s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:51.770484934 +0000 UTC m=+2976.218315388" watchObservedRunningTime="2026-09-17 18:23:51.771261008 +0000 UTC m=+2976.219091463"
Sep 17 18:23:52.753978 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:52.753938    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:52.754995 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:52.754964    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:23:53.758109 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:53.758065    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:23:58.763342 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:58.763297    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:23:58.763955 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:23:58.763930    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:24:08.764587 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:08.764546    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:24:17.919050 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:17.919020    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:24:17.927758 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:17.927734    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:24:18.764712 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:18.764673    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:24:28.764670 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:28.764635    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:24:38.764367 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:38.764314    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:24:46.650262 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:46.650223    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"]
Sep 17 18:24:46.650864 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:46.650451    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" containerID="cri-o://91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663" gracePeriod=30
Sep 17 18:24:46.650864 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:46.650485    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kube-rbac-proxy" containerID="cri-o://1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa" gracePeriod=30
Sep 17 18:24:47.001915 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:47.001838    2562 generic.go:356] "Generic (PLEG): container finished" podID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerID="1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa" exitCode=2
Sep 17 18:24:47.002061 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:47.001921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerDied","Data":"1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa"}
Sep 17 18:24:48.758771 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:48.758730    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 17 18:24:48.764102 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:48.764077    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 17 18:24:49.286806 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.286784    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:24:49.399420 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-rbac-proxy-sar-config\") pod \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") "
Sep 17 18:24:49.399420 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kserve-provision-location\") pod \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") "
Sep 17 18:24:49.399420 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-proxy-tls\") pod \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") "
Sep 17 18:24:49.399602 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399463    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-api-access-szwhq\") pod \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") "
Sep 17 18:24:49.399602 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-cabundle-cert\") pod \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\" (UID: \"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751\") "
Sep 17 18:24:49.399866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399834    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" (UID: "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:49.399986 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.399913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" (UID: "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:49.401434 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.401412    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" (UID: "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:49.401518 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.401452    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-api-access-szwhq" (OuterVolumeSpecName: "kube-api-access-szwhq") pod "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" (UID: "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751"). InnerVolumeSpecName "kube-api-access-szwhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:49.409198 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.409178    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" (UID: "307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:49.499972 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.499948    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:24:49.499972 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.499969    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwhq\" (UniqueName: \"kubernetes.io/projected/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-api-access-szwhq\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:24:49.500103 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.499977    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:24:49.500103 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.499986    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:24:49.500103 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:49.499994    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:24:50.016296 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.016217    2562 generic.go:356] "Generic (PLEG): container finished" podID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerID="91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663" exitCode=0
Sep 17 18:24:50.016296 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.016272    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"
Sep 17 18:24:50.016770 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.016297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerDied","Data":"91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663"}
Sep 17 18:24:50.016770 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.016368    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n" event={"ID":"307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751","Type":"ContainerDied","Data":"d74f705b874e41c3dfc425f16a73429b1b1d15b99aac825146c1f2d083973819"}
Sep 17 18:24:50.016770 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.016398    2562 scope.go:117] "RemoveContainer" containerID="1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa"
Sep 17 18:24:50.024808 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.024793    2562 scope.go:117] "RemoveContainer" containerID="91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663"
Sep 17 18:24:50.032116 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.032101    2562 scope.go:117] "RemoveContainer" containerID="9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744"
Sep 17 18:24:50.036764 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.036745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"]
Sep 17 18:24:50.040355 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.040307    2562 scope.go:117] "RemoveContainer" containerID="1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa"
Sep 17 18:24:50.040737 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:24:50.040704    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa\": container with ID starting with 1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa not found: ID does not exist" containerID="1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa"
Sep 17 18:24:50.040834 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.040749    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa"} err="failed to get container status \"1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa\": rpc error: code = NotFound desc = could not find container \"1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa\": container with ID starting with 1fff9473d06ff136f4df4a7a0d1dd1e4fefa3cd38ee6e5dceb45865057e392fa not found: ID does not exist"
Sep 17 18:24:50.040834 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.040770    2562 scope.go:117] "RemoveContainer" containerID="91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663"
Sep 17 18:24:50.041089 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:24:50.041071    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663\": container with ID starting with 91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663 not found: ID does not exist" containerID="91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663"
Sep 17 18:24:50.041131 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.041095    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663"} err="failed to get container status \"91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663\": rpc error: code = NotFound desc = could not find container \"91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663\": container with ID starting with 91383aee5341574001117fe8c30979f92fc1c4fc5a1888219e9565fb5d089663 not found: ID does not exist"
Sep 17 18:24:50.041131 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.041110    2562 scope.go:117] "RemoveContainer" containerID="9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744"
Sep 17 18:24:50.041267 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.041248    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ss69n"]
Sep 17 18:24:50.041449 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:24:50.041431    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744\": container with ID starting with 9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744 not found: ID does not exist" containerID="9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744"
Sep 17 18:24:50.041496 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:50.041456    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744"} err="failed to get container status \"9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744\": rpc error: code = NotFound desc = could not find container \"9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744\": container with ID starting with 9ad3a91370b5335ef20e081104df3c0923752963b7708cf424bb811a34708744 not found: ID does not exist"
Sep 17 18:24:51.727872 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.727837    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" path="/var/lib/kubelet/pods/307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751/volumes"
Sep 17 18:24:51.846588 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846552    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"]
Sep 17 18:24:51.846899 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846887    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kube-rbac-proxy"
Sep 17 18:24:51.846942 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846900    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kube-rbac-proxy"
Sep 17 18:24:51.846942 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846914    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container"
Sep 17 18:24:51.846942 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846919    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container"
Sep 17 18:24:51.846942 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846927    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="storage-initializer"
Sep 17 18:24:51.846942 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846932    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="storage-initializer"
Sep 17 18:24:51.847094 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846983    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kserve-container"
Sep 17 18:24:51.847094 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.846992    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="307eaa8f-edf7-4b6f-b2bf-5dff9fc4a751" containerName="kube-rbac-proxy"
Sep 17 18:24:51.850190 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.850174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:51.852778 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.852751    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:24:51.852915 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.852805    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:24:51.852915 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.852820    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:24:51.852915 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.852759    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:24:51.853742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.853694    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 18:24:51.853742 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.853730    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:24:51.853918 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.853730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 18:24:51.859872 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.859850    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"]
Sep 17 18:24:51.918006 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.917965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8842\" (UniqueName: \"kubernetes.io/projected/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-api-access-c8842\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:51.918183 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.918018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:51.918183 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.918083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:51.918300 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.918178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:51.918300 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:51.918231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.018919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.018963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019009 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.018999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8842\" (UniqueName: \"kubernetes.io/projected/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-api-access-c8842\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019284 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.019018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019284 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.019037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019284 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:24:52.019156    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 18:24:52.019284 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:24:52.019230    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls podName:175387e7-e4cc-4fc1-869c-dc66beb3a43c nodeName:}" failed. No retries permitted until 2026-09-17 18:24:52.519209519 +0000 UTC m=+3036.967039952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls") pod "isvc-pmml-predictor-8674795f59-fklbk" (UID: "175387e7-e4cc-4fc1-869c-dc66beb3a43c") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 18:24:52.019505 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.019412    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019826 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.019804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.019864 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.019809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.027935 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.027913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8842\" (UniqueName: \"kubernetes.io/projected/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-api-access-c8842\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.524540 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.524506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.526948 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.526925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-fklbk\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:52.762034 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:52.761997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:24:53.039398 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:53.039375    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"]
Sep 17 18:24:53.041628 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:24:53.041600    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175387e7_e4cc_4fc1_869c_dc66beb3a43c.slice/crio-7ae709f073d1900ada32253f4ce13f41ad88132ae07afa3a60b8387cb6360bf6 WatchSource:0}: Error finding container 7ae709f073d1900ada32253f4ce13f41ad88132ae07afa3a60b8387cb6360bf6: Status 404 returned error can't find the container with id 7ae709f073d1900ada32253f4ce13f41ad88132ae07afa3a60b8387cb6360bf6
Sep 17 18:24:54.040288 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:54.040253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerStarted","Data":"da895fbdcfad9bd51de07421973804175e8f67ae64c7b4b03fdc47a849ea56cf"}
Sep 17 18:24:54.040767 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:54.040319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerStarted","Data":"7ae709f073d1900ada32253f4ce13f41ad88132ae07afa3a60b8387cb6360bf6"}
Sep 17 18:24:57.054929 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:57.054895    2562 generic.go:356] "Generic (PLEG): container finished" podID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerID="da895fbdcfad9bd51de07421973804175e8f67ae64c7b4b03fdc47a849ea56cf" exitCode=0
Sep 17 18:24:57.055285 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:57.054975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerDied","Data":"da895fbdcfad9bd51de07421973804175e8f67ae64c7b4b03fdc47a849ea56cf"}
Sep 17 18:24:57.056201 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:24:57.056185    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:25:05.102854 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:05.102821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerStarted","Data":"2edff866d200a3edb660d29d49feb8c51b29446599bb2daaf9b2ddb1698d919e"}
Sep 17 18:25:05.103275 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:05.102883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerStarted","Data":"dfb9238ccee1bfc7eaadad75af7a081d807d1d4b60d62077360c1e085330a630"}
Sep 17 18:25:05.103275 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:05.103213    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:25:05.124822 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:05.124775    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podStartSLOduration=3.250910853 podStartE2EDuration="14.124761553s" totalImagesPullingTime="7.114347369s" totalInitContainerRuntime="3.759503331s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:05.123535137 +0000 UTC m=+3049.571365610" watchObservedRunningTime="2026-09-17 18:25:05.124761553 +0000 UTC m=+3049.572592005"
Sep 17 18:25:06.107884 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:06.107853    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:25:06.109158 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:06.109133    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:25:07.112775 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:07.112735    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:25:12.116722 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:12.116692    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:25:12.117187 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:12.117164    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:25:22.117664 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:22.117624    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:25:32.117625 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:32.117580    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:25:42.117268 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:42.117231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:25:52.117644 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:25:52.117610    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:26:02.117766 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:02.117724    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:26:12.117992 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:12.117946    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:26:22.118100 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:22.118057    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 18:26:32.118114 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:32.118043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:26:43.254250 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:43.254210    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"]
Sep 17 18:26:43.254721 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:43.254558    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container" containerID="cri-o://dfb9238ccee1bfc7eaadad75af7a081d807d1d4b60d62077360c1e085330a630" gracePeriod=30
Sep 17 18:26:43.254721 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:43.254593    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kube-rbac-proxy" containerID="cri-o://2edff866d200a3edb660d29d49feb8c51b29446599bb2daaf9b2ddb1698d919e" gracePeriod=30
Sep 17 18:26:43.557382 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:43.557283    2562 generic.go:356] "Generic (PLEG): container finished" podID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerID="2edff866d200a3edb660d29d49feb8c51b29446599bb2daaf9b2ddb1698d919e" exitCode=2
Sep 17 18:26:43.557382 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:43.557357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerDied","Data":"2edff866d200a3edb660d29d49feb8c51b29446599bb2daaf9b2ddb1698d919e"}
Sep 17 18:26:46.574468 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.574436    2562 generic.go:356] "Generic (PLEG): container finished" podID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerID="dfb9238ccee1bfc7eaadad75af7a081d807d1d4b60d62077360c1e085330a630" exitCode=0
Sep 17 18:26:46.574763 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.574510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerDied","Data":"dfb9238ccee1bfc7eaadad75af7a081d807d1d4b60d62077360c1e085330a630"}
Sep 17 18:26:46.691439 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.691419    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:26:46.845949 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.845913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls\") pod \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") "
Sep 17 18:26:46.846091 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.845970    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-rbac-proxy-sar-config\") pod \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") "
Sep 17 18:26:46.846091 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.845998    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kserve-provision-location\") pod \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") "
Sep 17 18:26:46.846091 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.846033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-cabundle-cert\") pod \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") "
Sep 17 18:26:46.846210 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.846096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8842\" (UniqueName: \"kubernetes.io/projected/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-api-access-c8842\") pod \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\" (UID: \"175387e7-e4cc-4fc1-869c-dc66beb3a43c\") "
Sep 17 18:26:46.846319 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.846274    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "175387e7-e4cc-4fc1-869c-dc66beb3a43c" (UID: "175387e7-e4cc-4fc1-869c-dc66beb3a43c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:46.846435 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.846410    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "175387e7-e4cc-4fc1-869c-dc66beb3a43c" (UID: "175387e7-e4cc-4fc1-869c-dc66beb3a43c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:46.846501 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.846460    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "175387e7-e4cc-4fc1-869c-dc66beb3a43c" (UID: "175387e7-e4cc-4fc1-869c-dc66beb3a43c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:46.847991 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.847974    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "175387e7-e4cc-4fc1-869c-dc66beb3a43c" (UID: "175387e7-e4cc-4fc1-869c-dc66beb3a43c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:46.848179 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.848158    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-api-access-c8842" (OuterVolumeSpecName: "kube-api-access-c8842") pod "175387e7-e4cc-4fc1-869c-dc66beb3a43c" (UID: "175387e7-e4cc-4fc1-869c-dc66beb3a43c"). InnerVolumeSpecName "kube-api-access-c8842". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:46.946588 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.946561    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8842\" (UniqueName: \"kubernetes.io/projected/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-api-access-c8842\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:26:46.946588 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.946586    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175387e7-e4cc-4fc1-869c-dc66beb3a43c-proxy-tls\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:26:46.946724 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.946601    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:26:46.946724 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.946613    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175387e7-e4cc-4fc1-869c-dc66beb3a43c-kserve-provision-location\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:26:46.946724 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:46.946626    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175387e7-e4cc-4fc1-869c-dc66beb3a43c-cabundle-cert\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:26:47.580886 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.580848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk" event={"ID":"175387e7-e4cc-4fc1-869c-dc66beb3a43c","Type":"ContainerDied","Data":"7ae709f073d1900ada32253f4ce13f41ad88132ae07afa3a60b8387cb6360bf6"}
Sep 17 18:26:47.580886 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.580880    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"
Sep 17 18:26:47.581371 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.580930    2562 scope.go:117] "RemoveContainer" containerID="2edff866d200a3edb660d29d49feb8c51b29446599bb2daaf9b2ddb1698d919e"
Sep 17 18:26:47.589683 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.589662    2562 scope.go:117] "RemoveContainer" containerID="dfb9238ccee1bfc7eaadad75af7a081d807d1d4b60d62077360c1e085330a630"
Sep 17 18:26:47.597112 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.597087    2562 scope.go:117] "RemoveContainer" containerID="da895fbdcfad9bd51de07421973804175e8f67ae64c7b4b03fdc47a849ea56cf"
Sep 17 18:26:47.602963 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.602938    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"]
Sep 17 18:26:47.607137 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.607118    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-fklbk"]
Sep 17 18:26:47.732620 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:26:47.732594    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" path="/var/lib/kubelet/pods/175387e7-e4cc-4fc1-869c-dc66beb3a43c/volumes"
Sep 17 18:29:17.949802 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:29:17.949773    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:29:17.959209 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:29:17.959187    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:29:18.103075 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:29:18.103042    2562 scope.go:117] "RemoveContainer" containerID="caefbf4111663a4eba8a3b1d15881802a52a8194ea6335f98e1d5e3b480dee7f"
Sep 17 18:29:18.113719 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:29:18.113698    2562 scope.go:117] "RemoveContainer" containerID="366c38b383d683cea01defe6bb449bc27ee485f155db356cddf4f8c17641023a"
Sep 17 18:29:18.121424 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:29:18.121403    2562 scope.go:117] "RemoveContainer" containerID="97c8f2e61742ef1bfb2a29a4d1d046b143fc8912d1f3550079407735e35b1087"
Sep 17 18:30:40.584080 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584043    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nk6z9/must-gather-bl8bl"]
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584426    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kube-rbac-proxy"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584442    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kube-rbac-proxy"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584453    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="storage-initializer"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584459    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="storage-initializer"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584470    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584476    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584531    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kube-rbac-proxy"
Sep 17 18:30:40.584573 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.584540    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="175387e7-e4cc-4fc1-869c-dc66beb3a43c" containerName="kserve-container"
Sep 17 18:30:40.587656 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.587635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.590486 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.590461    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nk6z9\"/\"kube-root-ca.crt\""
Sep 17 18:30:40.590605 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.590515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nk6z9\"/\"openshift-service-ca.crt\""
Sep 17 18:30:40.601256 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.601235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nk6z9/must-gather-bl8bl"]
Sep 17 18:30:40.648849 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.648822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vt5b\" (UniqueName: \"kubernetes.io/projected/76fda8e7-6ff9-4da7-a61c-21703811eadc-kube-api-access-4vt5b\") pod \"must-gather-bl8bl\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") " pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.649007 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.648879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76fda8e7-6ff9-4da7-a61c-21703811eadc-must-gather-output\") pod \"must-gather-bl8bl\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") " pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.750173 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.750143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vt5b\" (UniqueName: \"kubernetes.io/projected/76fda8e7-6ff9-4da7-a61c-21703811eadc-kube-api-access-4vt5b\") pod \"must-gather-bl8bl\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") " pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.750297 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.750193    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76fda8e7-6ff9-4da7-a61c-21703811eadc-must-gather-output\") pod \"must-gather-bl8bl\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") " pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.750521 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.750505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76fda8e7-6ff9-4da7-a61c-21703811eadc-must-gather-output\") pod \"must-gather-bl8bl\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") " pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.759405 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.759384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vt5b\" (UniqueName: \"kubernetes.io/projected/76fda8e7-6ff9-4da7-a61c-21703811eadc-kube-api-access-4vt5b\") pod \"must-gather-bl8bl\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") " pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:40.911244 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:40.911170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:30:41.181667 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:41.181642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nk6z9/must-gather-bl8bl"]
Sep 17 18:30:41.184571 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:30:41.184543    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76fda8e7_6ff9_4da7_a61c_21703811eadc.slice/crio-1907d20fdd80efc3f09ff0ff145b62b784664570d27814aa65fadeee9103b9f8 WatchSource:0}: Error finding container 1907d20fdd80efc3f09ff0ff145b62b784664570d27814aa65fadeee9103b9f8: Status 404 returned error can't find the container with id 1907d20fdd80efc3f09ff0ff145b62b784664570d27814aa65fadeee9103b9f8
Sep 17 18:30:41.186721 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:41.186703    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:30:41.633961 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:41.633926    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nk6z9/must-gather-bl8bl" event={"ID":"76fda8e7-6ff9-4da7-a61c-21703811eadc","Type":"ContainerStarted","Data":"1907d20fdd80efc3f09ff0ff145b62b784664570d27814aa65fadeee9103b9f8"}
Sep 17 18:30:47.676401 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:47.676363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nk6z9/must-gather-bl8bl" event={"ID":"76fda8e7-6ff9-4da7-a61c-21703811eadc","Type":"ContainerStarted","Data":"c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"}
Sep 17 18:30:48.682707 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:48.682670    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nk6z9/must-gather-bl8bl" event={"ID":"76fda8e7-6ff9-4da7-a61c-21703811eadc","Type":"ContainerStarted","Data":"10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f"}
Sep 17 18:30:48.702337 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:30:48.702269    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nk6z9/must-gather-bl8bl" podStartSLOduration=2.399177935 podStartE2EDuration="8.702249567s" totalImagesPullingTime="6.303071632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:48.701790796 +0000 UTC m=+3393.149621252" watchObservedRunningTime="2026-09-17 18:30:48.702249567 +0000 UTC m=+3393.150080023"
Sep 17 18:31:16.814712 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:16.814678    2562 generic.go:356] "Generic (PLEG): container finished" podID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerID="c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244" exitCode=0
Sep 17 18:31:16.815190 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:16.814741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nk6z9/must-gather-bl8bl" event={"ID":"76fda8e7-6ff9-4da7-a61c-21703811eadc","Type":"ContainerDied","Data":"c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"}
Sep 17 18:31:16.815190 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:16.815044    2562 scope.go:117] "RemoveContainer" containerID="c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"
Sep 17 18:31:17.219636 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.219548    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nk6z9_must-gather-bl8bl_76fda8e7-6ff9-4da7-a61c-21703811eadc/gather/0.log"
Sep 17 18:31:17.950045 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.950013    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5s492/must-gather-k99sq"]
Sep 17 18:31:17.953584 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.953568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:17.956057 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.956034    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s492\"/\"openshift-service-ca.crt\""
Sep 17 18:31:17.956189 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.956166    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s492\"/\"kube-root-ca.crt\""
Sep 17 18:31:17.957145 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.957129    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5s492\"/\"default-dockercfg-lrg5v\""
Sep 17 18:31:17.961598 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.961574    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s492/must-gather-k99sq"]
Sep 17 18:31:17.991577 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.991549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpg4\" (UniqueName: \"kubernetes.io/projected/3f2beab7-460b-4e10-b7c7-afdf123c80da-kube-api-access-xcpg4\") pod \"must-gather-k99sq\" (UID: \"3f2beab7-460b-4e10-b7c7-afdf123c80da\") " pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:17.991710 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:17.991594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f2beab7-460b-4e10-b7c7-afdf123c80da-must-gather-output\") pod \"must-gather-k99sq\" (UID: \"3f2beab7-460b-4e10-b7c7-afdf123c80da\") " pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:18.092887 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.092859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f2beab7-460b-4e10-b7c7-afdf123c80da-must-gather-output\") pod \"must-gather-k99sq\" (UID: \"3f2beab7-460b-4e10-b7c7-afdf123c80da\") " pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:18.093036 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.092922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpg4\" (UniqueName: \"kubernetes.io/projected/3f2beab7-460b-4e10-b7c7-afdf123c80da-kube-api-access-xcpg4\") pod \"must-gather-k99sq\" (UID: \"3f2beab7-460b-4e10-b7c7-afdf123c80da\") " pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:18.093183 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.093164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f2beab7-460b-4e10-b7c7-afdf123c80da-must-gather-output\") pod \"must-gather-k99sq\" (UID: \"3f2beab7-460b-4e10-b7c7-afdf123c80da\") " pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:18.103483 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.103464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpg4\" (UniqueName: \"kubernetes.io/projected/3f2beab7-460b-4e10-b7c7-afdf123c80da-kube-api-access-xcpg4\") pod \"must-gather-k99sq\" (UID: \"3f2beab7-460b-4e10-b7c7-afdf123c80da\") " pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:18.262990 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.262912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s492/must-gather-k99sq"
Sep 17 18:31:18.538457 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.538309    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s492/must-gather-k99sq"]
Sep 17 18:31:18.540802 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:31:18.540775    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f2beab7_460b_4e10_b7c7_afdf123c80da.slice/crio-7a8ad4fcb5903644ee0d5547cb66f6e5b6750e3697844defdef1593b30eee6d1 WatchSource:0}: Error finding container 7a8ad4fcb5903644ee0d5547cb66f6e5b6750e3697844defdef1593b30eee6d1: Status 404 returned error can't find the container with id 7a8ad4fcb5903644ee0d5547cb66f6e5b6750e3697844defdef1593b30eee6d1
Sep 17 18:31:18.825163 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:18.825129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/must-gather-k99sq" event={"ID":"3f2beab7-460b-4e10-b7c7-afdf123c80da","Type":"ContainerStarted","Data":"7a8ad4fcb5903644ee0d5547cb66f6e5b6750e3697844defdef1593b30eee6d1"}
Sep 17 18:31:19.835046 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:19.834997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/must-gather-k99sq" event={"ID":"3f2beab7-460b-4e10-b7c7-afdf123c80da","Type":"ContainerStarted","Data":"1c38d4f5fe4377f9065523569ad115db654ae898a90cfcc8a040861e247c05d2"}
Sep 17 18:31:19.835551 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:19.835225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/must-gather-k99sq" event={"ID":"3f2beab7-460b-4e10-b7c7-afdf123c80da","Type":"ContainerStarted","Data":"8a02656bf413b24ce359ff76c551a87f42cbdec16bbb95e7e01878e6ffff2179"}
Sep 17 18:31:19.855926 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:19.855873    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5s492/must-gather-k99sq" podStartSLOduration=1.9693970410000001 podStartE2EDuration="2.855856746s" totalImagesPullingTime="886.459705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:19.854415171 +0000 UTC m=+3424.302245626" watchObservedRunningTime="2026-09-17 18:31:19.855856746 +0000 UTC m=+3424.303687201"
Sep 17 18:31:20.964795 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:20.964765    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d2v8l_6c83b0a2-9af0-4601-9d20-79b3a6533dbf/global-pull-secret-syncer/0.log"
Sep 17 18:31:21.158228 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:21.158193    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wb25w_74b6bb2c-1e44-46bd-90c8-e6c1772d35f9/konnectivity-agent/0.log"
Sep 17 18:31:21.213504 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:21.213477    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-71.ec2.internal_4abd5f1ae2c9430a917972c6d312f06a/haproxy/0.log"
Sep 17 18:31:22.784227 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:22.784181    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-nk6z9/must-gather-bl8bl"]
Sep 17 18:31:22.784705 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:22.784486    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-nk6z9/must-gather-bl8bl" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="copy" containerID="cri-o://10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f" gracePeriod=2
Sep 17 18:31:22.788084 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:31:22.788053    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bl8bl\" is forbidden: User \"system:node:ip-10-0-133-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nk6z9\": no relationship found between node 'ip-10-0-133-71.ec2.internal' and this object" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:31:22.788250 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:22.788178    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-nk6z9/must-gather-bl8bl"]
Sep 17 18:31:23.174859 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.174818    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nk6z9_must-gather-bl8bl_76fda8e7-6ff9-4da7-a61c-21703811eadc/copy/0.log"
Sep 17 18:31:23.178083 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.178048    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:31:23.186354 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:31:23.183699    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bl8bl\" is forbidden: User \"system:node:ip-10-0-133-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nk6z9\": no relationship found between node 'ip-10-0-133-71.ec2.internal' and this object" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:31:23.242677 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.242639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76fda8e7-6ff9-4da7-a61c-21703811eadc-must-gather-output\") pod \"76fda8e7-6ff9-4da7-a61c-21703811eadc\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") "
Sep 17 18:31:23.243025 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.242993    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vt5b\" (UniqueName: \"kubernetes.io/projected/76fda8e7-6ff9-4da7-a61c-21703811eadc-kube-api-access-4vt5b\") pod \"76fda8e7-6ff9-4da7-a61c-21703811eadc\" (UID: \"76fda8e7-6ff9-4da7-a61c-21703811eadc\") "
Sep 17 18:31:23.246273 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.244698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fda8e7-6ff9-4da7-a61c-21703811eadc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "76fda8e7-6ff9-4da7-a61c-21703811eadc" (UID: "76fda8e7-6ff9-4da7-a61c-21703811eadc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:23.250480 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.250450    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fda8e7-6ff9-4da7-a61c-21703811eadc-kube-api-access-4vt5b" (OuterVolumeSpecName: "kube-api-access-4vt5b") pod "76fda8e7-6ff9-4da7-a61c-21703811eadc" (UID: "76fda8e7-6ff9-4da7-a61c-21703811eadc"). InnerVolumeSpecName "kube-api-access-4vt5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:23.347808 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.347755    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vt5b\" (UniqueName: \"kubernetes.io/projected/76fda8e7-6ff9-4da7-a61c-21703811eadc-kube-api-access-4vt5b\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.347808 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.347801    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/76fda8e7-6ff9-4da7-a61c-21703811eadc-must-gather-output\") on node \"ip-10-0-133-71.ec2.internal\" DevicePath \"\""
Sep 17 18:31:23.729958 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.729914    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" path="/var/lib/kubelet/pods/76fda8e7-6ff9-4da7-a61c-21703811eadc/volumes"
Sep 17 18:31:23.870494 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.870412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nk6z9_must-gather-bl8bl_76fda8e7-6ff9-4da7-a61c-21703811eadc/copy/0.log"
Sep 17 18:31:23.872006 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.871348    2562 generic.go:356] "Generic (PLEG): container finished" podID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerID="10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f" exitCode=143
Sep 17 18:31:23.872006 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.871418    2562 scope.go:117] "RemoveContainer" containerID="10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f"
Sep 17 18:31:23.872006 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.871578    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nk6z9/must-gather-bl8bl"
Sep 17 18:31:23.891918 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.891527    2562 scope.go:117] "RemoveContainer" containerID="c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"
Sep 17 18:31:23.911944 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.911615    2562 scope.go:117] "RemoveContainer" containerID="10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f"
Sep 17 18:31:23.912868 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:31:23.912181    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f\": container with ID starting with 10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f not found: ID does not exist" containerID="10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f"
Sep 17 18:31:23.912868 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.912216    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f"} err="failed to get container status \"10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f\": rpc error: code = NotFound desc = could not find container \"10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f\": container with ID starting with 10be02a13dfa38e01fc007b4156f65d02fe47abbecaa55484a35cfc8a559266f not found: ID does not exist"
Sep 17 18:31:23.912868 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.912241    2562 scope.go:117] "RemoveContainer" containerID="c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"
Sep 17 18:31:23.913223 ip-10-0-133-71 kubenswrapper[2562]: E0917 18:31:23.913140    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244\": container with ID starting with c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244 not found: ID does not exist" containerID="c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"
Sep 17 18:31:23.913223 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:23.913173    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244"} err="failed to get container status \"c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244\": rpc error: code = NotFound desc = could not find container \"c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244\": container with ID starting with c724e67723091d94940dd44a7d594d7d78c24275e043a1b5d61fee8f80359244 not found: ID does not exist"
Sep 17 18:31:25.415023 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.414993    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-5mc2q_902c01da-6632-47c1-80c1-c434756bdd6d/kube-state-metrics/0.log"
Sep 17 18:31:25.443734 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.443692    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-5mc2q_902c01da-6632-47c1-80c1-c434756bdd6d/kube-rbac-proxy-main/0.log"
Sep 17 18:31:25.471463 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.471430    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-5mc2q_902c01da-6632-47c1-80c1-c434756bdd6d/kube-rbac-proxy-self/0.log"
Sep 17 18:31:25.677003 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.676922    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tz26h_a0d72f17-d779-41b8-8450-0e6a76f53444/node-exporter/0.log"
Sep 17 18:31:25.704712 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.704660    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tz26h_a0d72f17-d779-41b8-8450-0e6a76f53444/kube-rbac-proxy/0.log"
Sep 17 18:31:25.731867 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.731842    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tz26h_a0d72f17-d779-41b8-8450-0e6a76f53444/init-textfile/0.log"
Sep 17 18:31:25.853798 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.853754    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-stqjh_1a0927c0-029c-459a-ada7-92c1ebc06c63/kube-rbac-proxy-main/0.log"
Sep 17 18:31:25.888547 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.888518    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-stqjh_1a0927c0-029c-459a-ada7-92c1ebc06c63/kube-rbac-proxy-self/0.log"
Sep 17 18:31:25.920477 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.920451    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-stqjh_1a0927c0-029c-459a-ada7-92c1ebc06c63/openshift-state-metrics/0.log"
Sep 17 18:31:25.965195 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.965124    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/prometheus/0.log"
Sep 17 18:31:25.984180 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:25.984151    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/config-reloader/0.log"
Sep 17 18:31:26.011064 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.011032    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/thanos-sidecar/0.log"
Sep 17 18:31:26.036290 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.035919    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/kube-rbac-proxy-web/0.log"
Sep 17 18:31:26.064429 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.064361    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/kube-rbac-proxy/0.log"
Sep 17 18:31:26.096502 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.096437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/kube-rbac-proxy-thanos/0.log"
Sep 17 18:31:26.125041 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.125011    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_64119c2b-c0e1-4d1c-b676-0518d0a1af4f/init-config-reloader/0.log"
Sep 17 18:31:26.158923 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.158885    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-cpfzx_b8eb0f76-cca0-4524-aba9-514e8edf0534/prometheus-operator/0.log"
Sep 17 18:31:26.183034 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.183000    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-cpfzx_b8eb0f76-cca0-4524-aba9-514e8edf0534/kube-rbac-proxy/0.log"
Sep 17 18:31:26.215866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.215784    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-92bpv_4570300d-7d11-49ce-8a0a-56523a07b61a/prometheus-operator-admission-webhook/0.log"
Sep 17 18:31:26.366290 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.366264    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578db865db-6wv6l_e175f2dd-3705-4b32-88ce-e9b6b3f014df/thanos-query/0.log"
Sep 17 18:31:26.401242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.401215    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578db865db-6wv6l_e175f2dd-3705-4b32-88ce-e9b6b3f014df/kube-rbac-proxy-web/0.log"
Sep 17 18:31:26.426595 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.426562    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578db865db-6wv6l_e175f2dd-3705-4b32-88ce-e9b6b3f014df/kube-rbac-proxy/0.log"
Sep 17 18:31:26.455208 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.455132    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578db865db-6wv6l_e175f2dd-3705-4b32-88ce-e9b6b3f014df/prom-label-proxy/0.log"
Sep 17 18:31:26.480764 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.480662    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578db865db-6wv6l_e175f2dd-3705-4b32-88ce-e9b6b3f014df/kube-rbac-proxy-rules/0.log"
Sep 17 18:31:26.503436 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:26.503408    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578db865db-6wv6l_e175f2dd-3705-4b32-88ce-e9b6b3f014df/kube-rbac-proxy-metrics/0.log"
Sep 17 18:31:27.888370 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:27.888339    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-8lqdp_d916b8ba-958f-4cb4-aadc-ab6f3467ac17/networking-console-plugin/0.log"
Sep 17 18:31:28.469233 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:28.469203    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/2.log"
Sep 17 18:31:28.474640 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:28.474613    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-fn7x9_7de942b3-1609-4be9-9632-b39a80a5d381/console-operator/3.log"
Sep 17 18:31:29.554364 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.554286    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"]
Sep 17 18:31:29.554939 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.554868    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="gather"
Sep 17 18:31:29.554939 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.554888    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="gather"
Sep 17 18:31:29.554939 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.554921    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="copy"
Sep 17 18:31:29.554939 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.554930    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="copy"
Sep 17 18:31:29.555139 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.555021    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="copy"
Sep 17 18:31:29.555139 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.555033    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fda8e7-6ff9-4da7-a61c-21703811eadc" containerName="gather"
Sep 17 18:31:29.559560 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.559538    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.568319 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.568288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"]
Sep 17 18:31:29.607776 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.607742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-proc\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.607776 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.607778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-podres\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.608021 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.607847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-sys\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.608021 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.607873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-lib-modules\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.608021 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.607960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tzk\" (UniqueName: \"kubernetes.io/projected/f9533631-9dde-46cc-93ef-eb015d13e8e6-kube-api-access-g9tzk\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.708880 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.708836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-proc\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709065 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.708892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-podres\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709065 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.708940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-sys\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709065 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.708966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-lib-modules\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709065 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.709035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tzk\" (UniqueName: \"kubernetes.io/projected/f9533631-9dde-46cc-93ef-eb015d13e8e6-kube-api-access-g9tzk\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709507 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.709483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-proc\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709685 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.709552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-sys\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709685 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.709606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-podres\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.709838 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.709609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9533631-9dde-46cc-93ef-eb015d13e8e6-lib-modules\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.718713 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.718686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tzk\" (UniqueName: \"kubernetes.io/projected/f9533631-9dde-46cc-93ef-eb015d13e8e6-kube-api-access-g9tzk\") pod \"perf-node-gather-daemonset-jtjh7\" (UID: \"f9533631-9dde-46cc-93ef-eb015d13e8e6\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:29.878559 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:29.878520    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:30.165644 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.165611    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"]
Sep 17 18:31:30.169587 ip-10-0-133-71 kubenswrapper[2562]: W0917 18:31:30.169534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9533631_9dde_46cc_93ef_eb015d13e8e6.slice/crio-bb76246e3d52eb17f8a2851095cc64f79769cf771d13a14e9a04aab9979553af WatchSource:0}: Error finding container bb76246e3d52eb17f8a2851095cc64f79769cf771d13a14e9a04aab9979553af: Status 404 returned error can't find the container with id bb76246e3d52eb17f8a2851095cc64f79769cf771d13a14e9a04aab9979553af
Sep 17 18:31:30.583250 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.583169    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bljxr_c86541f5-fa09-4a99-b838-01322d9c0cb6/dns/0.log"
Sep 17 18:31:30.610168 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.610141    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bljxr_c86541f5-fa09-4a99-b838-01322d9c0cb6/kube-rbac-proxy/0.log"
Sep 17 18:31:30.815346 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.815286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mwgsw_6ebdeed2-06ee-41de-b637-0ae050cf8c63/dns-node-resolver/0.log"
Sep 17 18:31:30.916148 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.916045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7" event={"ID":"f9533631-9dde-46cc-93ef-eb015d13e8e6","Type":"ContainerStarted","Data":"d15226ede479af8ecd1ba662eeddf60ba8ec35598ce74a7844425ba5c34a0938"}
Sep 17 18:31:30.916148 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.916114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7" event={"ID":"f9533631-9dde-46cc-93ef-eb015d13e8e6","Type":"ContainerStarted","Data":"bb76246e3d52eb17f8a2851095cc64f79769cf771d13a14e9a04aab9979553af"}
Sep 17 18:31:30.916458 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.916234    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:30.937167 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:30.937119    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7" podStartSLOduration=1.9371035239999999 podStartE2EDuration="1.937103524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:30.935907913 +0000 UTC m=+3435.383738369" watchObservedRunningTime="2026-09-17 18:31:30.937103524 +0000 UTC m=+3435.384933979"
Sep 17 18:31:31.340649 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:31.340615    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j9mf9_a9557f36-6cf8-46fb-b997-9f9ade8915be/node-ca/0.log"
Sep 17 18:31:32.514419 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:32.514385    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7swlh_04876813-f72f-4194-8dab-62304008b825/serve-healthcheck-canary/0.log"
Sep 17 18:31:32.933194 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:32.933171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54hqg_34f19014-b54a-417b-a27d-8211a23fb72e/kube-rbac-proxy/0.log"
Sep 17 18:31:32.959295 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:32.959271    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54hqg_34f19014-b54a-417b-a27d-8211a23fb72e/exporter/0.log"
Sep 17 18:31:32.986321 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:32.986293    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-54hqg_34f19014-b54a-417b-a27d-8211a23fb72e/extractor/0.log"
Sep 17 18:31:36.158665 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.158637    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jgrrc_db08db2e-17e4-426c-89d4-ae8ab7d03b89/s3-init/0.log"
Sep 17 18:31:36.175844 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.175816    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jgrrc_db08db2e-17e4-426c-89d4-ae8ab7d03b89/download-hf-model/0.log"
Sep 17 18:31:36.206721 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.206698    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-929h4_d47ab0bf-69e5-4414-aad5-9f69615c1ebc/s3-tls-init-custom/0.log"
Sep 17 18:31:36.230282 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.230254    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-929h4_d47ab0bf-69e5-4414-aad5-9f69615c1ebc/download-hf-model/0.log"
Sep 17 18:31:36.265779 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.265756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-pn8gk_8ecb4956-c0a8-4a11-899b-75904645726e/s3-tls-init-serving/0.log"
Sep 17 18:31:36.286826 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.286805    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-pn8gk_8ecb4956-c0a8-4a11-899b-75904645726e/download-hf-model/0.log"
Sep 17 18:31:36.321998 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.321972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-4ncs4_1e104153-ae7f-42f9-a8b7-7401a1ee2a4f/seaweedfs/0.log"
Sep 17 18:31:36.349174 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.349146    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-22ff4_22478116-1143-4463-90d3-2baf7abe1142/seaweedfs-tls-custom/0.log"
Sep 17 18:31:36.930559 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:36.930529    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jtjh7"
Sep 17 18:31:41.421262 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:41.421231    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-bjwcr_8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13/kube-storage-version-migrator-operator/1.log"
Sep 17 18:31:41.422634 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:41.422616    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-bjwcr_8dbb2901-fd04-4f6f-a6c6-65b4f61dbb13/kube-storage-version-migrator-operator/0.log"
Sep 17 18:31:42.706247 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.706212    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:31:42.731468 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.731442    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/egress-router-binary-copy/0.log"
Sep 17 18:31:42.756375 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.756323    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/cni-plugins/0.log"
Sep 17 18:31:42.782765 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.782741    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/bond-cni-plugin/0.log"
Sep 17 18:31:42.839964 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.839932    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/routeoverride-cni/0.log"
Sep 17 18:31:42.861108 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.861081    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/whereabouts-cni-bincopy/0.log"
Sep 17 18:31:42.885243 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.885215    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs8n_59ba2826-88d8-472b-8c1d-2f9d06841f6d/whereabouts-cni/0.log"
Sep 17 18:31:42.988054 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:42.987974    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fz8ls_1e806d7b-ef85-4912-912f-428b60ebc744/kube-multus/0.log"
Sep 17 18:31:43.148233 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:43.148203    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rzpqd_fbc60926-841c-4824-872b-30a7c806ade0/network-metrics-daemon/0.log"
Sep 17 18:31:43.170077 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:43.170051    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rzpqd_fbc60926-841c-4824-872b-30a7c806ade0/kube-rbac-proxy/0.log"
Sep 17 18:31:43.977531 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:43.977502    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/ovn-controller/0.log"
Sep 17 18:31:43.995458 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:43.995432    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/ovn-acl-logging/0.log"
Sep 17 18:31:44.030784 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:44.030757    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/kube-rbac-proxy-node/0.log"
Sep 17 18:31:44.057242 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:44.057198    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:31:44.076774 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:44.076746    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/northd/0.log"
Sep 17 18:31:44.101771 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:44.101741    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/nbdb/0.log"
Sep 17 18:31:44.131960 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:44.131933    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/sbdb/0.log"
Sep 17 18:31:44.202214 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:44.202180    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5l8m4_7cde33cc-1dab-4ddc-9c17-1ff58962ee01/ovnkube-controller/0.log"
Sep 17 18:31:45.691769 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:45.691738    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-dp2xf_2e211d54-8b6d-4ae0-b999-ab1a3b22ed1e/check-endpoints/0.log"
Sep 17 18:31:45.743866 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:45.743837    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7c8nf_0fa66574-a870-4c4a-9932-e850183522b4/network-check-target-container/0.log"
Sep 17 18:31:46.715748 ip-10-0-133-71 kubenswrapper[2562]: I0917 18:31:46.715720    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-b9dts_a5c1752e-456c-417e-9ef1-25e6efc078d1/iptables-alerter/0.log"